In the measurement practice of gear helix deviation, there is phenomenon that big difference often appears between helix deviations for different gear teeth. In order to improve the measurement accuracy of gear helix deviation, influence laws of installation error of mandrel and gear on gear helix deviation were respectively researched. Firstly, mathematical models of influences of mandrel installation eccentricity and tilt error as well as gear installation eccentricity and deflection error on gear helix deviation were respectively established. Then precision test experiment of gear helix deviation was conducted by using manufactured flat washer (1#, 4#) and tilt washers with tilt errors respectively of 5.5 μm/45 mm (2#) and 11.9 μm/45 mm (3#). The following result was achieved: the difference of maximum for helix slope deviation fHβ and theoretical model was 0.17 μm and relative error was 7% to adopt 2# tilt washer; the difference of maximum for helix slope deviation fHβ and theoretical model was 0.06μm and relative error was 1% to adopt 3# tilt washer; while shape deviation ffβ of gear helix was basically fixed in the two tests. Experimental result indicates that measured result of deflection error of gear installation on helix deviation is basically the same as theoretical value to verify accuracy of established mathematical model. Error compensation method of compensating helix slope deviation difference through adjusting deflection error of gear installation according to established mathematical model is proposed, which is of important research meaning to develop high precision gear artifact.