Combining electric field distribution of Gaussian beam and plane-wave analysis method, the photo-electric signal analytical expression of homodyne vibrometer system irradiated by Gaussian beam was deduced. And relative measurement error caused by Gaussian beam was analyzed. The relationships between relative measurement error and the distance from detector to laser beam transmission axis, the distance from detector to beam-waist, vibration displacement and beam-waist radius were discussed, respectively. The homodyne vibrometer system of specific parameters was simulated numerically and the simulation results coincided with the theoretical analysis. The results show that the relationship between measurement error and vibration displacement and beam-waist radius is complex. Measurement error increases with the increase of vibration displacement while fluctuating cyclically. There’s a zero point and maxima in the relationship curve of beam-waist radius and measurement error. It provides a theoretical basis for error analysis in high-precision laser measurement.