To further suppress the effects of assembly error disturbance on the output performance of a fast steering mirror control system and to improve the tracking accuracy of the system, a disturbance suppression method with internal and external disturbance observation links based on the principle of disturbance observation is proposed. The effects of the assembly error of the fast steering mirror on the accuracy of the control system were analyzed, and an equivalent mathematical model of the unbalance torque disturbance was developed. The inner loop interference observation link was used to suppress medium- and high-frequency disturbances, whereas the outer loop interference observation link was used to compensate for the medium- and low-frequency disturbance amplification of the inner loop interference observation link and provide medium- and low-frequency disturbance suppression. The input and error signal transmission process of the proposed disturbance suppression method were analyzed, and the control parameters of each controller of the suppression system were designed. A virtual prototype simulation test platform was built to test the performance of the control system. The results show that the adjustment time error of the virtual prototype system is 0.54% before and after the double interference observation link is added. For a 15-Hz equivalent disturbance, the disturbance suppression capability is improved by 25.23%. Theoretical and experimental results show that the introduction of a double disturbance observation links can effectively suppress the disturbance of the assembly error.