On the basis of the technological application of modulation transfer spectroscopy, absorption frequency discrimination, and dispersion frequency discrimination, according to the operating theory of passive hydrogen maser, the theoretical analysis of the process where a single frequency modulation signal carries out frequency discrimination of hydrogen transition and of the microwave cavity resonance are mathematically derived in details. The relation of frequency discrimination curves and the error signal amplitudes with different modulation depths from simulation is obtained by simulation, and the simulated results are compared with the experimental results. The simulated and experimental results verify the effectiveness of the derivation process, and the modulation depth is acquired, which endows the short-term stability of passive hydrogen maser with the best performance. The analysis procedure provides the theoretical basis of performance optimization for passive hydrogen maser based on single frequency modulation, and it provides the design principle of project improvement for electronic circuit.