In view of the problems of the loss of detailed information caused by wavelet transform, the non-sparsity of low-frequency subband coefficient decomposed by Non-Subsampled Contourlet Transform (NSCT), and the poor comprehensive performance of infrared and visible image fusion, an algorithm for fusion of infrared and visible images is proposed based on sparse representation, NSCT, and Pulse Coupled Neural Network (PCNN). Firstly, the original image is decomposed by NSCT to obtain the low-frequency and high-frequency subbands.Secondly, the K-SVD (Singular Value Decomposition) algorithm is used to carry out dictionary training on the low-frequency subband to realize the sparse representation of low-frequency subband and the fusion of low-frequency sparse coefficients. Then, the spatial frequency of the high-frequency subband is utilized to stimulate PCNN, and the coefficient with more ignition times is selected as the fusion coefficient of the high-frequency subband. Finally, the NSCT inverse transform is applied to the low and high frequency subband fusion coefficients to obtain the fused image.The experimental results show that the proposed algorithm has a great advantage in subjective visual effect and objective index evaluation, and its comprehensive performance is superior to that of the existing algorithm.