The radial motion error is one of the most important errors of rotary axis, which seriously affects the precision and performance of CNC machine tools. Using laser interference with reference rotary axis can achieve measurement of radial motion error. Neither target or reference rotary axis is ideal, there are some restrictions like uneven speed, response delay and unstable tracking. These restrictions cause phase jitter and make the non-linear correction difficult. It will affect the phase calculation accuracy of the interference signal. In order to eliminate the influence of phase jitters, a type of signal processing method is proposed. The paper designs and builds a set of laser interference measuring device for measuring the radial motion error. Compared to the traditional correction method, the proposed method reduces the measurement repeatability from 4.8 μm to 0.2 μm, and the error compared with the standard instrument is decreased from 3.5 μm to 2 μm.