To enhance the measurement accuracy of gear size and shape errors, a model of the influence of the eccentricity and tilt errors of the reference axis on the measurement of gears and the evaluated accuracy of gears is established for the research object of a line scan gear measuring machine, which has the difficult of calibrating the reference axis of rotary table. Based on this, an improved method for calibrating reference axis based on monocular vision is proposed. Fixing the checkerboard grid parallel to the turntable, using the camera to collect the feature point data, using the least squares method to fit the spatial plane and center of the circle of the feature point, and finally completing the reference axis calibration.The experimental results show that compared with the results of the calibration before the improved method, the tilted angle between the calibrated axis and the standard reference axis of the calibration after the improved method is reduced by 65.227 1% and the value of eccentricity is reduced by 47.875 2%.The error of the improved calibration results is significantly reduced and the calibration results are more accurate. Compared with other calibration methods, the calibration method is simple to operate.