Due to the nonlinear motion of the scanning platform in the actual measurement system, the random scanning error is introduced, which leads to the limitation of absolute precision and measurement repeatability. In this study, a random scanning error suppression method is proposed to solve this problem. The principle of mutual calibration of the second-order slope of the theoretical modulation curves and the second-order slope of the actual modulation curves of the grating fringe is used to achieve the accurate determination of the scanning step. The feasibility and progressiveness of this method are proved by theoretical derivation, simulation and experimental verification, which demonstrates that the measurement repeatability standard deviation is better than 1 nm in simulation results, the absolute precision is better than 3 nm and the measurement repeatability standard deviation is better than 2 nm in experiments.