A hybrid quantum system is constracted based on the traditional cavity optomechanical system coupled with the surface acoustic wave (SAW) resonator, and then the physical mechanism of the nonreciprocal phonon-photon quantum interface is studied using this model. The results show that the essential reason to realize this nonreciprocal quantum information transfer is the interference effect, and the coupling strength and phase between the mechanical resonator and the auxiliary optical cavity play a key role during the manipulation of nonreciprocal information.