Photo-elastic modulator (PEM) is a high-Q resonant device. When driven by a high voltage resonant signal, its resonant frequency will drift with its temperature change, and the stability of the interferometer and the accuracy of the rebuilt spectrum are affected. The vibration model and frequency- drifting model of the photo-elastic modulator were first established, and a frequency-tracking and adjustable-amplitude double closed-loop driving control method was proposed. In the method the digital phase-locking technique was adopted to track the changing resonant frequency of the photo-elastic modulator. Based on the change of the maximum optical path difference for the reference laser interferometer, the amplitude of the driving signal was adjusted to realize the stable control of the rebuilt spectrum. At the same time, the calibration of the reconstructed spectra was realized based on the maximum optical path difference. In the experiment the double closed-loop adaptive drive control method was applied to the driving control system of the photo-elastic modulated Fourier transform spectrometer (PEM-FTs), which realized the real-time tracking of the driving-frequency to the resonant frequency for PEM and the amplitude adjustment of the high-voltage amplifier. The maximum optical path difference of the interference diagram was stable at about 0.236 nm, and its precision was 3.3%. The maximum relative error of the reconstructed spectra was 2.5%. The experiment shows that the method can effectively stabilize the spectral resolution of the PEM-FTs.