Infrared and Laser Engineering, Volume. 51, Issue 12, 20220125(2022)
A review of deep learning fusion methods for infrared and visible images
Figures & Tables(19)
![Infrared and visible image fusion framework based on convolutional neural network[43]](/Images/icon/loading.gif){kind=icon}
Fig. 1. Infrared and visible image fusion framework based on convolutional neural network[43]
Fig. 2. Infrared and visible image fusion framework based on autoencoder network[54]
Fig. 3. Infrared and visible image fusion framework based on generative countermeasure network[10]
Fig. 4. Infrared and visible images collected by Athena, DHV, FEL and TRICLOBS systems respectively. (a1)-(d1) Infrared images; (a2)-(d2) Visible images
Fig. 5. The Images in roads, vehicles, and pedestrians scenes respectively. (a1)-(c1) Infrared images; (a2)-(c2) Visible images
Fig. 6. The images in BackyardRunner, CoatDeposit, GroupFight, and MulitpleDeposit scenes respectively. (a1)-(d1) Infrared images; (a2)-(d2) Visible images
Fig. 7. The first frames in two video sequences respectively. (a1), (b1) Infrared images; (a2), (b2) Visible image
Fig. 8. Qualitative fusion results. (a), (b) Infrared and visible images; (c)-(l) Fusion methods of DenseFuse, FusionDN, U2 Fusion, FusionGAN, DDcGAN, GANMcC, RFN-Nest, STDFusionNet, SDDGAN and SeAFusion
Table 1. Limitations of typical CNN-based fusion methods
View tableView in ArticleTable 1. Limitations of typical CNN-based fusion methods
References Limitation [ 40 ]Being suitable for mutil-focus image fusion, only the last convolutional layer features are used to calculate the fusion result [ 46 ]The information in the middle layer is lost, and the fusion strategy has no theoretical support [ 50 ]The structure is simple and prone to overfitting [ 54 ]The model mainly saves detailed texture information and cannot highlight infrared targets
Table 2. Limitations of typical autoencoder-based fusion methods
View tableView in ArticleTable 2. Limitations of typical autoencoder-based fusion methods
References Limitation [ 57 ]The model is not targeted enough to highlight the infrared target, and the fusion strategy is simple [ 64 ]Insufficient attention to texture information, large amount of network parameters are not conducive to application [ 68 ]Network channels share weights, pre-training models focus on common information, and unique information may be lost [ 67 ]Abundant texture details cannot be obtained
Table 3. Limitations of typical GAN-based fusion methods
View tableView in ArticleTable 3. Limitations of typical GAN-based fusion methods
References Limitation [ 10 ]Insufficient consideration of infrared brightness information. [ 79 ]The two adversarial losses are difficult to balance, and fusion image target is distorted [ 86 ]Under the two-discriminator condition, the Wasserstein distance adversarial loss does not enhance the target brightness [ 89 ]The lack of well-segmented datasets, the quality of the pre-training model depends on the accuracy of semantic segmentation
Table 4. Summary of infrared and visible image fusion methods based on deep learning
View tableView in ArticleTable 4. Summary of infrared and visible image fusion methods based on deep learning
Type Typical methods Characteristic Input method Single channel [ 10 ], [45 ], [52 ], [54 -55 ], [79 ], [82 ], [83 ]Cascading source images, mining the fusion ability of the network Multi-channel [ 46 ], [47 ], [50 -51 ], [53 ], [56 -72 ]Distinguishing the source images, but need to design a fusion strategy Multi-image multi-channel [ 67 ], [88 -89 ]Inputting the source images in proportion, keeping the same category information of the source images Preprocess image [ 70 -71 ], [86 ], [89 -90 ]Providing more useful information for fused images Common block Attention network [ 45 ], [51 ], [53 ], [63 ], [65 ], [85 ], [87 ]Enhancing feature maps from channels and spaces, it can be embedded in any network Nest network [ 63 -65 ]The network structure is complex, and focusing on the shallow and middle layers of the network Skip connection [ 59 ], [68 ], [77 ], [87 ]Based on residual and dense networks，it prevents loss of useful shallow information Loss Func-tion Perceptual loss [ 55 ], [66 ], [82 ], [87 ]Balancing feature error between reconstructed image and input TV loss [ 47 ], [79 ]constraining the fused image to exhibit similar gradient variation with the visible image Edge detail loss [ 69 ], [82 ], [83 -84 ]Enhancing fusion image edge detail Sematic loss [ 72 ]More targeted to different information of the scene
Table 5. Objective evaluation indicators of different methods in the Kaptein_1654 scene
View tableView in ArticleTable 5. Objective evaluation indicators of different methods in the Kaptein_1654 scene
Methods EN MI SSIM SD AG SF PSNR VIFF CC QAB/F DenseFuse 6.42 12.83 0.72 29.74 3.62 6.93 16.39 0.34 0.53 0.36 FusionDN 7.19 14.37 0.64 46.48 6.67 12.97 14.56 0.55 0.52 0.42 U2 Fusion 6.58 13.16 0.70 28.68 4.61 8.62 16.17 0.35 0.53 0.40 FusionGAN 5.74 11.47 0.67 17.10 3.29 6.28 17.05 0.08 0.64 0.17 DDcGAN 6.96 13.93 0.59 37.17 6.29 11.63 15.15 0.32 0.52 0.38 GANMcC 6.06 12.11 0.69 25.36 2.13 4.44 15.38 0.21 0.56 0.14 RFN-Nest 6.54 13.09 0.68 31.47 2.39 4.99 15.69 0.32 0.52 0.28 STDFusionNet 6.70 13.41 0.65 52.90 5.29 11.22 15.17 0.40 0.51 0.54 SSDGAN 5.85 11.70 0.64 23.06 1.49 3.66 13.15 0.19 0.56 0.08 SeAFusion 6.71 13.43 0.67 41.07 5.86 11.25 13.92 0.41 0.56 0.49
Table 6. Objective evaluation indicators of different methods in the Sandpath scenario
View tableView in ArticleTable 6. Objective evaluation indicators of different methods in the Sandpath scenario
Methods EN MI SSIM SD AG SF PSNR VIFF CC QAB/F DenseFuse 6.68 13.35 0.69 29.06 6.61 10.89 19.86 0.54 0.68 0.36 FusionDN 7.42 14.84 0.55 45.13 10.95 18.35 15.75 0.82 0.68 0.29 U2 Fusion 6.34 12.67 0.69 20.46 6.42 10.49 19.19 0.36 0.69 0.33 FusionGAN 6.43 12.85 0.60 21.12 6.22 10.30 17.05 0.13 0.66 0.29 DDcGAN 7.25 14.50 0.49 37.85 10.23 16.91 15.03 0.44 0.69 0.37 GANMcC 6.33 12.65 0.69 21.12 3.40 5.66 19.44 0.22 0.71 0.19 RFN-Nest 6.89 13.79 0.64 32.01 4.88 8.13 19.24 0.50 0.68 0.42 STDFusionNet 6.82 13.64 0.59 35.09 6.83 11.64 18.59 0.22 0.60 0.56 SSDGAN 5.95 11.91 0.62 16.26 2.08 3.56 16.32 0.18 0.71 0.09 SeAFusion 6.82 13.64 0.65 33.02 7.43 12.26 17.81 0.35 0.66 0.42
Table 7. Objective evaluation indicators of different methods in the campus_1 scenario
View tableView in ArticleTable 7. Objective evaluation indicators of different methods in the campus_1 scenario
Methods EN MI SSIM SD AG SF PSNR VIFF CC QAB/F DenseFuse 7.13 14.26 0.64 37.50 7.32 17.07 14.99 0.29 0.87 0.44 FusionDN 7.56 15.12 0.60 50.79 11.32 25.36 14.55 0.33 0.86 0.42 U2 Fusion 7.16 14.32 0.62 37.79 9.04 19.94 14.95 0.30 0.88 0.40 FusionGAN 6.15 12.29 0.55 18.65 5.56 13.03 12.98 0.08 1.08 0.14 DDcGAN 7.38 14.76 0.53 44.16 11.48 24.23 14.19 0.23 0.83 0.38 GANMcC 7.16 14.33 0.59 36.89 6.21 11.02 15.49 0.21 0.88 0.21 RFN-Nest 7.19 14.39 0.58 39.47 5.04 11.19 14.83 0.27 0.87 0.22 STDFusionNet 7.39 14.79 0.68 49.13 11.30 28.52 15.67 0.17 0.85 0.50 SSDGAN 6.70 13.40 0.52 30.07 3.46 8.11 13.56 0.18 0.95 0.12 SeAFusion 7.56 15.11 0.59 53.36 11.82 27.87 14.21 0.29 0.86 0.47
Table 8. Objective evaluation indicators of different methods in the campus_2 scenario
View tableView in ArticleTable 8. Objective evaluation indicators of different methods in the campus_2 scenario
Methods EN MI SSIM SD AG SF PSNR VIFF CC QAB/F DenseFuse 7.41 14.81 0.62 50.76 8.76 20.71 15.13 0.50 0.93 0.44 FusionDN 7.44 14.88 0.60 51.99 10.33 23.62 14.85 0.47 0.91 0.46 U2 Fusion 7.32 14.64 0.60 52.49 10.57 23.99 15.05 0.52 0.92 0.50 FusionGAN 6.70 13.40 0.49 27.51 6.02 14.02 12.23 0.23 0.97 0.16 DDcGAN 7.36 14.72 0.52 46.74 10.12 23.09 14.16 0.28 0.90 0.37 GANMcC 7.42 14.84 0.55 48.77 6.21 13.88 15.53 0.45 0.93 0.26 RFN-Nest 7.42 14.84 0.55 49.85 6.07 13.71 14.80 0.45 0.94 0.26 STDFusionNet 7.34 14.68 0.58 54.27 11.33 28.35 13.99 0.32 0.88 0.49 SSDGAN 7.02 14.03 0.46 42.71 4.89 11.78 13.70 0.35 0.93 0.17 SeAFusion 7.71 15.42 0.60 66.00 13.79 32.00 13.50 0.48 0.56 0.92
Table 9. Objective evaluation indicators of different methods in the MulitpleDeposit scenario
View tableView in ArticleTable 9. Objective evaluation indicators of different methods in the MulitpleDeposit scenario
Methods EN MI SSIM SD AG SF PSNR VIFF CC QAB/F DenseFuse 7.66 15.31 0.78 71.02 6.65 15.86 16.64 0.72 0.96 0.55 FusionDN 7.46 14.91 0.75 53.44 7.22 16.87 16.61 0.59 0.95 0.52 U2 Fusion 7.23 14.64 0.77 52.49 10.57 23.99 15.05 0.52 0.97 0.50 FusionGAN 7.13 14.27 0.70 43.06 5.98 14.21 15.88 0.28 0.97 0.38 DDcGAN 7.29 14.58 0.67 47.60 6.85 15.47 15.74 0.36 0.95 0.43 GANMcC 7.71 15.42 0.75 73.35 4.44 10.38 15.27 0.57 0.96 0.36 RFN-Nest 7.70 15.39 0.75 72.45 4.96 11.86 16.14 0.67 0.99 0.47 STDFusionNet 7.50 15.01 0.72 72.25 8.86 23.66 19.84 0.69 0.92 0.62 SSDGAN 7.67 15.35 0.71 67.96 3.77 8.58 15.90 0.51 0.96 0.25 SeAFusion 7.79 15.59 0.74 76.36 8.80 21.46 17.97 0.80 0.95 0.62
Table 10. Objective evaluation indicators of different methods in the VisitorParking scenario
View tableView in ArticleTable 10. Objective evaluation indicators of different methods in the VisitorParking scenario
Methods EN MI SSIM SD AG SF PSNR VIFF CC QAB/F DenseFuse 6.77 13.54 0.74 34.64 4.41 10.97 18.18 0.51 0.67 0.42 FusionDN 7.50 15.00 0.63 52.26 7.92 19.62 14.32 0.70 0.63 0.40 U2 Fusion 6.53 13.05 0.74 31.37 4.55 11.30 19.06 0.41 0.66 0.40 FusionGAN 6.19 12.39 0.65 32.06 3.90 10.00 14.59 0.21 0.63 0.27 DDcGAN 7.26 14.52 0.61 42.80 7.02 16.99 14.84 0.55 0.66 0.39 GANMcC 6.76 13.53 0.72 39.09 2.76 6.60 18.88 0.37 0.65 0.22 RFN-Nest 7.18 14.36 0.68 44.54 3.57 9.41 15.22 0.61 0.66 0.40 STDFusionNet 6.28 12.55 0.68 26.09 4.83 14.43 20.30 0.19 0.61 0.49 SSDGAN 6.49 12.97 0.72 29.54 2.09 5.01 19.27 0.34 0.67 0.13 SeAFusion 6.79 13.58 0.70 35.86 5.96 14.84 16.46 0.47 0.66 0.47
Table 11. Running time of different methods
View tableView in ArticleTable 11. Running time of different methods
Methods Run time/s DenseFuse 3.7234 FusionDN 2.5158 U2 Fusion 1.0212 FusionGAN 0.5221 DDcGAN 2.4545 GANMcC 1.0142 RFN-Nest 1.1682 STDFusionNet 0.0480 SDDGAN 0.1970 SeAFusion 0.1605
Tools
Get Citation
Copy Citation Text
Lin Li, Hongmei Wang, Chenkai Li. A review of deep learning fusion methods for infrared and visible images[J]. Infrared and Laser Engineering, 2022, 51(12): 20220125
Paper Information
Category: Image processing
Received: Feb. 23, 2022
Accepted: --
Published Online: Jan. 10, 2023
The Author Email:
© Copyright 2018-2021 | Chinese Laser Press.
All Rights Reserved 沪ICP备15018463号-20