Infrared and visible light image fusion is an enhancement technique designed to create a fused image that retains the advantages of the source image. In this study, a depth image decomposition-based infrared and visible image fusion method is proposed. First, the source image is decomposed into the background feature map and detail feature map by the encoder; simultaneously, the saliency feature extraction module is introduced in the encoder to highlight the edge and texture features of the source image; subsequently, the fused image is obtained by the decoder. In the training process, a gradient coefficient penalty was applied to the visible image for regularized reconstruction to ensure texture consistency, and a loss function was designed for image decomposition and reconstruction to reduce the differences between the background feature maps and amplify the differences between the detail feature maps. The experimental results show that the method can generate fused images with rich details and bright targets. In addition, this method outperforms other comparative methods in terms of subjective and objective evaluations of the TNO and FLIR public datasets.