Being affected by birefringence, the polarization state of single photon is randomly and slowly altered during the transmission in single mode fiber. This influences the performance of quantum key distribution system. The “plug and play” system with Faraday rotation mirror is theoretically analyzed by the use of probability wave superposition. The results show that the bidirectional transmission of a single photon through the same fiber can effectively offset the influence of fiber birefringence, but is vulnerable to nonreciprocity effects such as magneto-optical effect. The probability of normal interference fringes, formed by the single photon reaching the interferometer successfully, decreases with the increase of magneto-optical rotation angle. When the rotation angle is greater than 42°, the probability of forming the normal interference fringes is close to 0.