Phase Generated Carrier (PGC) demodulation algorithm has been widely used in laser coherent vibration measurement and gained widespread interest due to its high accuracy, large dynamic range, good linearity, and low hardware overhead. However, the PGC demodulation technology is always accompanied by nonlinear distortions induced by phase modulation depth deviation, light intensity disturbance, carrier phase delay, etc. Therefore, an improved PGC demodulation algorithm is urgently required, which can effectively suppress the nonlinear distortions.In this study, we propose an improved PGC demodulation algorithm based on low frequency modulation and iteratively reweighted ellipse specific fitting, which suppresses the nonlinear distortions in the laser vibration measurement. The ellipse specific fitting is realized by introducing a 6×6 ellipse constraint matrix in the direct least square fitting of ellipse, which avoids getting a hyperbola solution, consequently. The iteratively reweighted ellipse specific fitting uses iteratively reweighted optimization technology to improve the precision of the ellipse specific fitting and reduce the weight of the outlier data, it has the advantages of ellipse-specificity, high robustness and high precison. In the imrpoved demodulation algorithm, the iteratively reweighted ellipse specific fitting is used to correct the original quadrature signals into a pair of perfect quadrature signals, which eliminates the nonlinear errors. Furthermore, to overcome the drawback of the ellipse fitting algorithm that it fails to work correctly under small phase signals, a low frequency modulation with a large amplitude is added in the carrier modulation and it guarantees the ellipse fitting accuracy regarless of the desired signal amplitudes. Finally, differential cross multiplying is used to extract the desired phase shift signal from the corrected quadrature signals.The simulations of ellipse specific fiting and iteratively reweighted ellipse specific fitting are performed and the results show that the iteratively reweighted ellipse specific fitting is superior. Then the proposed algorithm is verified in a Michelson inteferometer and the experimental results show that the Lissajous figure of the quadrature signals without a stimulus is observed to be a 1/4 ellipse arc, a 1/2 ellipse arc, a 3/4 ellipse arc and a full ellipse when the amplitude of the low frequency modulation is set as 0.035 V, 0.085 V, 0.013 5 V, and 0.185 V, respectively. Then, a 1 kHz stimulus with the amplitude of 100 mV is set, it is found that the fitted Lissajous figure deviates from the standard circle when there is no low frequency modulation while it overlaps well with the circle when the low frequency modulation amplitude is larger than 0.085 V. Thus, the accuracy of the ellipse fitting results can be gurantted by introducing a appropriate low frequency modulation. The frequency spectra of the demodulated signals under the low frequency modulation of 0 V, 0.085 V, 0.0135 V, and 0.185 V are compared, nonlinear distortions are well supressed when the low amplitude is larger than 0.085 V. The demodulation algorithms based on ellipse sepcfific fitting and iteratively reweighted ellipse specific fitting are also compared in the experiment, the Signal-to-Noise-And Distortion ratio (SINAD) and Total Harmonic Distortion (THD) of the demodulated signal based on iteratively reweighted ellipse specific fitting are improved by 1.99 dB and 0.27%, respectively. The demodulated signals of the improved algorithm at the phase modulation depth range of 0.8~3.4 rad show a high stability, the mean SINAD and THD are 42.99 dB and 0.44% with the corresponding standard deviations of 0.55 dB and 0.03%, respectively. The stimulating response linearity of the system is better than 99.99% and the dynamic range reaches 103.90 dB @ 500 Hz & THD=1%. The operating frequency band of the system is 20~8 000 Hz and two vibration signals are successfully demodulated in the experiment.The improved PGC demodulation algorithm has a promising application prospect in the field of laser vibration measurement because of the advantages of high precision, good linearity, strong robustness and high computational efficiency.