In order to solve the problem that the information from the source images is easy to interfere with each other which influences the quality of infrared and visible image fusion, a new image fusion method based on Guided filter, Gaussian filter and nonsubsampled directional filter bank was proposed. The low-frequency approximation components, strong edge components and high-frequency detail components were obtained by combining Guided and Gaussian filter. Then the high-frequency detail components were filtered to obtain the detail directional components with the use of nonsubsampled directional bank. The low-frequency approximation components were fused by a fusion rule based on regional energy and the strong edge components were fused by a strategy based on convolutional sparse representation. The detail directional components were fused by a rule based on improved pulse coupled neural network. Then the final fused results were obtained by using inverse transform through fusing the fused components. Experimental results show that the proposed algorithm outperforms traditional methods in terms of visual inspection and objective measures. Compared with the image fusion algorithm based on discrete wavelet transform and sparse representation, which possesses the better fusion effect in the traditional methods, the fusion quality indexes of the proposed method, such as Standard deviation(STD), Information entropy(IE), Mutual information(MI), Average gradient (AG) and Spatial frequency(SF) increased by 20.28%, 2.24%, 47.41%, 5.34%, 8.02% averagely.