In this paper, an interference phenomenon and its elimination method in Fourier infrared imaging spectral data are analyzed and studied. In Fourier infrared imaging spectrometer, in some cases, a sharp convex interference peak appears in the spectral data. The peak position of this kind of interference peak is not fixed, and the amplitude will change greatly. The phenomenon from infrared detector, atmospheric absorption characteristics, target radiation characteristics, system noise and so on are analyzed. The reason for this phenomenon is the gray fluctuation of infrared detector. In view of the formation mechanism of the interference, the noise fitting method is proposed to eliminate the interference. The experimental results show that the proposed method can effectively eliminate the interference peak in Fourier infrared imaging spectrometer, and improve the detection effect and anti-interference ability of Fourier infrared imaging spectrometer.