To compensate the nonlinear errors of heterodyne interferometers in nanometer measurement, an experimental investigation is conducted.Based on the polarization properties of a coated corner-cube retro-reflector, a model is derived for describing the effect of the axial rotation of a measuring cube-corner retro-reflector in the motion direction on the first-harmonic nonlinearity when a laser emits the elliptical polarized light.The simulation results indicate that the first-harmonic nonlinearity can be reduced by the axial rotation of the cube-corner retro reflector.The experimental result shows that the first harmonic nonlinearity is reduced from 3.48 to 1.39 nm when the axial orientation rotation angle of the measuring cube-corner retro reflector increases from 0° to 100°,which is 40% that of the original one.The method simplies the complexes of light path and circuit systems for nonlinear error compensation approaches.