To improve the processing performance of source images with noise and improve the contrast and structural information of the fused images, an infrared and visible image fusion algorithm based on three-scale decomposition and sparse representation is proposed. First, to enhance the noise removal effect and maintain the structure and edge characteristics of the source image, the rolling guidance filter is used to decompose the source image, and the source image is decomposed into the base layer and the detail layer. Secondly, in order to make full use of the details and energy in the base component and reduce the complexity of the model, a structure-texture decomposition model is constructed. The base layer is decomposed into the base structure layer and base texture layer again, Then by analyzing the different characteristics of the three components, different fusion rules are utilized to fuse the three components respectively. The detail component contains the main noise components, but the noise level is different. Therefore, the sparse fusion denoising parameter is adaptively determined according to the noise level of the image to realize the fusion and denoising of the detail component at the same time and can effectively improve computational efficiency; for the base structure components, which contain fewer detail features, the weighted average technology based on the visual saliency map is directly used for pre-fusion; for the base texture components, because they contain visually important information or image features, such as edges, lines and contours and other activity information, which can reflect the main details of the original base image, the principal component analysis method is used for pre-fusion. Finally, the fused image is obtained by reconstructing the detail, base structure and texture layers. In order to verify the effectiveness of the proposed method, several groups of infrared and visible images were selected for experiments and compared with five recent methods, including CNN, FPDE, ResNet, IFEVIP and TIF, and the results were analyzed in a subjective and objective form. The experimental results show that, compared with other image fusion algorithms, this method can consider the fusion of noise-perturbed and noise-free images and can retain the details, brightness and structure of the source image in the fusion image with or without noise. Moreover, can effectively eliminate noise.