The arctangent demodulation compensation algorithm with parameter estimation error was proposed through analyzing laser speech detection system in theory, then quantitative mathematical formulas between the total harmonic distortion(THD) and key parameters as well as errors were built up. The THD and spurious free dynamic range(SFDR) were considered in this paper, which indicated the demodulation effect of micro vibration signal and minimum signal value distinguished from a large disturbance signal (jamming signal), respectively. This system implemented non-contact, long distance and high sensitive acoustic micro vibration signal detection. Laboratory experiments and simulations were presented to verify the feasibility of the arctangent demodulation compensation algorithm. Results illustrate that this system can detect the micro vibration caused by sound wave and recover voice signal from received signal within 35 meters using the compensation algorithm based on optical hardware platform currently.