A novel fusion method of infrared and visible images was proposed based on Non-subsampled Dual-tree Complex Contourlet Transform(NSDTCT) and sparse representation to overcome the shortcomings of traditional image fusion method based on wavelet transform. With the proposed method, morphological transform was used to deal with source images, and then the source images were decomposed by the NSDTCT to obtain the low frequency sub-band coefficients and high frequency sub-band coefficients. According to the different characteristics of the low and high frequency coefficients, an Improved Sparse Representation (ISR) fusion rule was proposed for the low frequency sub-bands; Then, the improved spatial frequency was used as the external input of a pulse coupled neural network, and a fusion method based on the improved adaptive dual channel pulse coupled neural network (2APCNN) was presented for the high frequency sub-bands. Finally, the fused image was obtained by performing the inverse NSDTCT. Experimental results indicate that the proposed method outperforms the conventional image fusion methods in terms of both objective evaluation criteria and visual quality. As compared with conventional NSCT-SR method, the fusion quality indexes, Mutual Information(MI), Mount of edge Information(QAB/F ), Average Gradient(AG) and Standard Deviation(SD) have increased by 9.89%, 6.39%, 104.64%, 55.09%, and 953%, 17.77%, 95.66%, 52.89%, respectively.