The Doppler asymmetric spatial heterodyne interferometer, a new type of mid- and upper-atmospheric wind-field detection system, can achieve atmospheric wind-field measurement by the inversion of the Doppler shift of observed source spectra after calculating the changes in interferograms. The reference phase is a necessary parameter to determine the Doppler shift of the wind field, and its stability is one of the core indicators to ensure the accuracy of wind speed measurement. This paper investigates three factors that affect the reference phase of an interferometer, namely, the phase drift of asymmetric quantities, phase slope drift, and phase drift of interferograms. Moreover, the theoretical analysis of the thermal phase drift is carried out on the basis of the principle of Doppler asymmetric spatial heterodyne interference. The separating and testing method for the phase-drift quantities of each factor is proposed, and the experimental test is conducted by the near-infrared Doppler asymmetric spatial heterodyne interferometer. Under the ambient temperature fluctuation of 0.27 ℃, the change of phase slope is 670 mrad/m, and the phase-drift fluctuation range of interferograms is 8.9 mrad. Upon the phase-drift correction of interferograms, the phase drift of asymmetric quantities is about 4.7 mrad, and the root mean square is 0.98 mrad, with the equivalent wind speed measurement error of 0.81 m/s. According to the bias experiment on temperature, the rate of phase-drift change of asymmetric quantities with temperature is -493 mrad/℃.