Based on the imaging characteristics of the spatially modulated imaging interferometer, the new compression method for interference spectral images, weighted rate-distortion optimization for set partitioning in hierarchical trees (SPIHT), is proposed. According to the image features in time and frequency domain, it is deduced by the proposed method that, with the increase of the optical path difference in time domain, the effect of interference spectral images on the spectral curve profile in frequency domain is increasing. In the optical path difference direction, the slope ratio of rate distortion is lifted increasingly, and the protection for images is enhanced, which not only makes up the deficiency of code rate distribution in SPIHT, but also protects the spectral information in frequency domain efficiently. The experimental results show that the proposed algorithm achieves improved performance over the conventional algorithm in time and frequency domain.