Acta Optica Sinica, Volume. 45, Issue 11, 1110001(2025)
Semantic Information Driven Multimodal Image Fusion Network
Figures & Tables(11)
Fig. 5. Visual quality comparison on TNO dataset. (a) VIS; (b) IR; (c) DenseFuse; (d) RFN-Net; (e) FusionGAN; (f) SDNet; (g) U2Fusion; (h) SeAFusion; (i) PIAFusion; (j) ours
Fig. 6. Visual quality comparison on MSRS dataset. (a) VIS; (b) IR; (c) DenseFuse; (d) RFN-Net; (e) FusionGAN; (f) SDNet; (g) U2Fusion; (h) SeAFusion; (i) PIAFusion; (j) ours
Fig. 7. Visual quality comparison of important loss functions and module ablation studies. (a) VIS; (b) IR; (c) MEGB; (d) MEGB+Sobel; (e) MEGB*+SBM; (f) MEGB*+SDRB; (g) MEGB*+SPC-Net; (h) MEGB*+SBM+SDRB; (i) ours
Fig. 8. Visual quality comparison of semantic segmentation results. (a) VIS; (b) IR; (c) ours; (d) ground truth
Table 1. Performance comparison of different methods across multiple datasets
View tableTable 1. Performance comparison of different methods across multiple datasets
Dataset Algorithm MI VIF /bit AG SCD EN /bit QAB/F TNO DenseFuse 2.1408 0.6704 2.4895 1.5916 6.3422 0.2486 RFN-Nest 1.4428 0.8103 2.6109 1.7711 6.9285 0.2262 FusionGAN 2.2010 0.6457 2.3636 1.3688 6.5199 0.2405 SDNet 2.3162 0.7523 4.5252 1.5488 6.6670 0.4484 U2Fusion 2.4808 0.6787 3.4891 1.5862 6.4230 0.3272 SeAFusion 2.4048 0.7986 3.2772 1.7172 6.6307 0.5295 PIAFusion 3.4884 0.8835 4.4265 1.6540 6.8937 0.4496 Ours 3.9817 0.8621 5.5097 1.8117 7.0620 0.6217 MSRS DenseFuse 2.1428 0.6694 2.6528 1.5051 6.4264 0.2486 RFN-Nest 1.4328 0.7338 2.5848 1.6352 6.7151 0.2262 FusionGAN 2.2510 0.5154 2.3610 1.1257 6.4690 0.2405 SDNet 2.3966 0.6231 4.0228 1.3912 6.6134 0.4484 U2Fusion 2.4107 0.7061 3.8500 1.5488 6.6285 0.3272 SeAFusion 3.6917 0.6969 2.1329 1.4934 6.5734 0.5496 PIAFusion 3.7317 0.9300 4.9702 1.3363 6.8036 0.6295 Ours 4.5717 0.8594 5.4374 1.7589 6.9482 0.6817
Table 2. Quantitative Evaluation Results of Ablation Study
View tableTable 2. Quantitative Evaluation Results of Ablation Study
Group MEGB Sobel SBM SDRB SPC-Net MI VIF /bit AG SCD EN /bit QAB/F 1 √ 2.2458 0.4782 3.1330 0.7154 5.0242 0.3887 2 √ √ 2.3727 0.5957 3.3583 1.2744 5.6838 0.3625 3 √ √ √ 2.3115 0.6486 3.3681 1.4740 5.7327 0.3415 4 √ √ √ 2.4162 0.6764 3.1844 1.5672 5.9284 0.4648 5 √ √ √ 2.5874 0.6719 2.9179 1.5780 5.8847 0.4783 6 √ √ √ √ 3.6884 0.7354 3.2551 1.6536 6.1237 0.5531 7 √ √ √ √ √ 3.9048 0.7854 3.3791 1.8057 6.4044 0.6495
Table 3. Segmentation performance of VIS, IR, and fused images at different times in the same scene
View tableTable 3. Segmentation performance of VIS, IR, and fused images at different times in the same scene
Label class Background Car Person Bike Curve Car stop Guardrail Color tone Bump Mean Day VIS 0.9800 0.8906 0.5556 0.7260 0.5798 0.4824 0.8090 0.6508 0.5669 0.6934 IR 0.9482 0.5470 0.6564 0.0847 0.1032 0.1268 0.0368 0.0087 0.1304 0.2936 Ours 0.9834 0.9074 0.7332 0.7347 0.5469 0.5395 0.7588 0.6335 0.5534 0.7101 Night VIS 0.9652 0.6960 0.1305 0.5889 0.2750 0.1762 0.3666 0.3792 0.1943 0.4191 IR 0.9593 0.4680 0.7103 0.0873 0.2599 0.0292 0.0000 0.0223 0.1945 0.3034 Ours 0.9763 0.7902 0.7205 0.6057 0.4419 0.2881 0.3390 0.4354 0.2233 0.5356 All VIS 0.9723 0.7933 0.3431 0.6575 0.4274 0.3293 0.5878 0.5150 0.3806 0.5563 IR 0.9538 0.5075 0.6834 0.0860 0.1816 0.0780 0.0184 0.0155 0.1625 0.2985 Ours 0.9799 0.8488 0.7269 0.6702 0.4944 0.4138 0.5489 0.5345 0.3884 0.6229
Tools
Get Citation
Copy Citation Text
Yulan Han, Yaozu Zhai, Tong Wu, Chaofeng Lan. Semantic Information Driven Multimodal Image Fusion Network[J]. Acta Optica Sinica, 2025, 45(11): 1110001
Paper Information
Category: Image Processing
Received: Jan. 26, 2025
Accepted: Apr. 15, 2025
Published Online: Jun. 23, 2025
The Author Email: Yulan Han (hanyulan@hrbust.edu.cn)
CSTR:32393.14.AOS250551
© Copyright 2018-2021 | Chinese Laser Press.
All Rights Reserved 沪ICP备15018463号-20