When larger aperture holographic plane gratings are fabricated by the holographic exposure method, the homodyne frequency-shifting interference pattern phase locking system usually is used to improve the holographic exposure quality. This paper focuses on the ultra-precision control of the system. Firstly, the principles and structures of a novel homodyne frequency-shifting interference pattern phase locking system were introduced. The high-order linear model was fitted for a nonlinear model of the system based on the theoretical modeling and system identification and a controller of the system was designed by combining the fitting model and the system vibration test result. Then, an actual controlling test was performed based on the design result of the controller to implement the ultra-precision control of the system. Finally, an analysis method on frequency domain was used to explore the effect factors on control precision for the ripple wave in the system. The experiment results demonstrate that the controlling error of the system has reached to ±0.046 1 rad(3σ) and it shows high frequency error ripple wave. The causes of the high frequency error were researched, and the results indicate that limited to the noise, delay of the system and controller parameters, the drift of the interference pattern caused by continuous micro-vibration with a wide frequency range can not be completely restrained by the controller.