Considering that infrared polarization and intensity image contain common information and their own unique information, a method of image fusion based on Dual-Tree Complex Wavelet Transform and sparse representation was proposed. Firstly, the high and low frequency components of source images wereobtained by using Dual-Tree Complex Wavelet Transform, and the high frequency components werecombined by absolute maximum method. Secondly, a joint matrix was constructed by low frequency components, and a redundant dictionarywas acquired by using K-singular value decomposition to train the matrix. Based on the dictionary, the sparse coefficient of low frequency component was calculated, and the common information and unique information werejudged by the location of non-zero value of the sparse coefficient, and two kinds of information was merged by proper fusion rules. Finally, the fusion image was obtained by performing inverse Dual-Tree Complex Wavelet Transform on the fused high and low frequency components. The experimental results show that the proposed fusion method can highlight the common information of source images and keep their own unique information, and the fusion image own higher contrast and clearer details.