In order to meet the test and analysis requirements of axis rotation error for the axis that cannot rotate complete circle, a new method is researched on the basis of the mathematical modeling track analysis. At first, the measurement error model caused by the pose errors of plane mirror and collimator for axis rotation error test is derived by using the quaternion method. The pose errors of plane mirror and collimator will cause an ellipse track on the collimator screen, while the ellipse will degenerate into a standard circle when the errors are small. Then, the fitting method of track parameters, including the center and radius, is established based on the least square principle to remove the influences of the pose errors on the test results. Finally, the effectiveness of the method proposed in this paper is verified by simulations and experiments. The experimental results indicate that the track parameters method has obvious advantages in the axis rotation error test for the axes that cannot rotate a complete circle and it can be used as a powerful supplement to the traditional Fourier series method. Besides the method needs not to fine adjust the plane mirror and collimator in the test, increasing the test and analysis efficiency.