The mechanism which suppresses Q-switched tendency by self-frequency-doubled (SFD) effect in a SFD mode-locked laser with a semiconductor saturable absorbed mirror (SESAM) was theoretically studied and numerically simulated. The expression of nonlinear losses, including the saturable loss of SESAM and the nonlinear loss caused by SFD effect, was deduced. The partial differential equations for intracavity pulse energy were developed. The stable conditions of the mode-locked laser were theoretically studied by the perturbation method. It was proved that suitable SFD effect can suppress Q-switching instabilities and reduce the threshold of stable mode-locking. Numerical simulations of mode-locked laser with and without SFD effect were carried out, which certificated the theoretical analysis.