To overcome the hysteresis of a tip/tilt mirror(TTM) in an adaptive optical system and to improve its linearity and control precision, this paper researches the nonlinear effects of the TTM. A compensation method based on the frequency-dependent Mutified-Prandtl-Ishlinskii (MPI) model is proposed for online adaptive inverse compensation of the hysteresis nonlinearity of the TTM. By combining with PID feedback control algorithm, an adaptive inverse feedforward composite control algorithm is proposed. In which, the adaptive inverse feedforward overcomes the hysteresis curve changes due to the frequency factors, and the PID algorithm improves the overall control performance. A second-order model of the TTM is established to estimate the output of the TTM system, which solves the reference signals of the MPI model, avoids adding extra feedforward sensors, and ensures the utilization of light energies. Experiment results indicate that this algorithm effectively solves the problem of hysteresis nonlinearity of the TTM, the 15 Hz hysteresis ratio of the TTM fells from 24.28% to 1.17%,and the linearity is improved by about 95%. As compared with the traditional PID algorithm, the control precision has increased by about 60%. It concludes that this composite control algorithm can compensate the hysteresis nonlinearity of the TTM effectively, and the control precision of the TTM is improved in adaptive optical systems.