In this study, we propose an infrared and visible image fusion algorithm based on semantic loss, to ensure that the generated images contain more semantic information through semantic loss, thereby satisfying the requirements of advanced vision tasks. First, a pre-trained segmentation network is used to segment the fused image, with the segmentation result and label map determining the semantic loss. Under the joint guidance of semantic and content losses, we force the fusion network to guarantee the quality of the fused image by considering the amount of semantic information in the image, to ensure that the fused image meets the requirements of advanced vision tasks. In addition, a new feature extraction module is designed in this study to achieve feature reuse through a residual dense connection to improve detail description capability while further reducing the fusion framework, which improves the time efficiency of image fusion. The experimental results show that the proposed algorithm outperforms existing fusion algorithms in terms of subjective visual effects and quantitative metrics and that the fused images contain richer semantic information.