In order to better mine and retain the details of the original image and improve the performance of image fusion, an infrared and visible image fusion algorithm based on chrominance loss in high exposure scenes was introduced, in which an auto-encoder structure was adopted, and it was composed of encoder and decoder. In the encoder part, an high exposure suppression module (EXBlock) for adaptive luminance detection was proposed to suppress the interference of high exposure to scene details, which could effectively balance the global contrast of images. In the decoder part, in order to realize the fusion of deep and shallow features, the multi-scale feature reconstruction method was used to realize the complementary and enhanced features in different scales. In the network training stage, in order to reduce the suppression of local color information in the high-exposure scene, the chrominance loss was designed and introduced to guide the fusion image to converge to the color information, so as to realize the color retention of the fusion image in the high-exposure scene. In order to verify the effectiveness of the algorithm, the subjective evaluation such as fusion result map and objective evaluation such as visual fidelity and mutual information was presented respectively in the experimental part. Compared with other algorithms, the proposed algorithm performs better and can better retain the chrominance information of the original images.