The voice coil motor-driven fast steering mirror is an important part of a high-precision photoelectric tracking system. In the photoelectric tracking system of the moving platform, the fast steering mirror system will suffer more complex and intense internal and external interference. The traditional passive interference suppression methods and the active interference suppression methods that treat the interference as lumped interference will not be enough to ensure the high-precision stability of boresight. Therefore, this paper proposes a sliding mode composite layered interference observation and compensation control strategy which combines harmonic interference observation and extended state observation. Firstly, the harmonic disturbance observer is used to observe the harmonic disturbance with a priori frequency information. Then the extended state observer is used to observe other unknown disturbances. Finally, based on the observed multi-source interference, the sliding mode nonlinear method with anti-interference ability is used to design a composite controller to maximize the suppression of multi-source disturbances suffered by the system. The experiment shows that the sliding mode composite layered interference observation compensation method proposed in this paper can significantly improve the LOS stability accuracy of the fast steering mirror compared with the traditional single interference observation compensation method.