Image fusion methods based on deep learning have achieved excellent image fusion performance and have been widely used in biometric recognition, automatic driving and target tracking. However, it is still challenging to extract important texture details and preserve information of images. Therefore, a loss function for infrared and visible image fusion networks is presented. We employ the histogram of oriented gradient (HOG) to calculate the loss function. HOG feature can reflect the direction and magnitude of local gradient in the image, and using HOG feature as the loss function can improve the ability of the network to extract image details. We combine HOG loss with multi-scale structural similarity loss, and train NestFuse, Res2Fusion and UNFusion infrared and visible image fusion networks with the designed loss function. On the TNO dataset, our model increases the standard deviation (SD) of fused images by 2.1476%, 1.2273% and 1.4444% respectively, and increases the visual information fiedity (VIF) of fused images by 1.6529%, 1.4936% and 1.2902% respectively. On the RoadScene dataset, our model increases the SD of the fused images by 1.0083%, 1.1669% and 0.7214% respectively, and increases the VIF of the fused images by 1.8093%, 1.8063% and 1.0406% respectively. The experimental results show that the proposed loss function can extract more effective information from the source image.