Soft Histogram of Gradients Loss: A loss Function for Optimization of the Image Fusion Networks

Yuxin Long; Wenjie Lai; Huaiyuan Zhang; Hongbo Zhang; Chengshi Li; Ziji Liu

doi:10.3788/LOP230882

Laser & Optoelectronics Progress, Volume. 60, Issue 24, 2411001(2023)

Soft Histogram of Gradients Loss: A loss Function for Optimization of the Image Fusion Networks

Yuxin Long, Wenjie Lai, Huaiyuan Zhang, Hongbo Zhang, Chengshi Li, and Ziji Liu^*

College of Optoelectronic Science and Engineering, University of Electronic Science and Technology, Chengdu 611731, Sichuan, China

show less

Abstract Get PDF(in Chinese)

Figures & Tables(11)

Fig. 1. Structure of three training networks . (a) Structure of NestFuse ^[3]; (b) structure of Res2Fusion^[7]; (c) structure of UNFusion^[8]

Download full size

Fig. 2. Image comparison fused by different networks on the TNO dataset. (a) (b) Source images; (c) NestFuse; (d) NestFuse-HOG; (e) Res2Fusion; (f) Res2Fusion-HOG; (g) UNFusion; (h) UNFusion-HOG

Download full size

Fig. 3. Image comparison fused by different networks on the RoadScene dataset. (a) (b) Source images; (c) NestFuse; (d) NestFuse-HOG; (e) Res2Fusion; (f) Res2Fusion-HOG; (g) UNFusion; (h) UNFusion-HOG

Download full size

Fig. 4. Fusion image comparison before and after loss function applied in NestFuse network. (a) (b) Source image 1; (c) image of source image 1 fused by NestFuse network; (d) image of source image 1 fused by NestFuse-HOG network; (e) (f) source image 2; (g) image of source image 2 fused by NestFuse network; (h) image of source image 2 fused by NestFuse-HOG network

Download full size

Fig. 5. Image comparison fused by different networks on the TNO dataset. (a) (b) Source images; (c) NestFuse; (d) NestFuse-grad; (e) NestFuse+grad; (f) NestFuse-perceptual; (g) NestFuse-HOG

Download full size

Fig. 6. Image comparison fused by different networks on the RoadScene dataset. (a) (b) Source images; (c) NestFuse; (d) NestFuse-grad; (e) NestFuse+grad; (f) NestFuse-perceptual; (g) NestFuse-HOG

Download full size

Table 1. Original loss function of the network
View table
Table 1. Original loss function of the network
Network Image intensity loss Structural similarity loss
NestFuse MSE MS-SSIM
Res2Fusion MSE MS-SSIM

Table 2. Quantitative comparison on the TNO dataset

View table

Table 2. Quantitative comparison on the TNO dataset

Network	EN	SD	MI	SCD	MS-SSIM	$F M I_d c t$	$F M I_w$	$Q_{a b f}$	VIF
NestFuse	6.9198	82.7523	13.8397	1.7335	0.8625	0.3580	0.4371	0.4870	0.7865
NestFuse-HOG	6.9435	84.5295	13.8870	1.7491	0.8684	0.3426	0.4323	0.4892	0.7995
Res2Fusion	6.7963	77.6068	13.5927	1.7365	0.8681	0.3745	0.4373	0.4760	0.7164
Res2Fusion-HOG	6.8087	78.5593	13.6173	1.7398	0.8682	0.3721	0.4362	0.4760	0.7271
UNFusion	6.9070	83.6454	13.8139	1.7097	0.8505	0.3506	0.4380	0.4940	0.8061
UNFusion-HOG	6.9210	84.8536	13.8419	1.7131	0.8500	0.3486	0.4412	0.4964	0.8165

Table 3. Quantitative comparison on the RoadScene dataset

View table

Table 3. Quantitative comparison on the RoadScene dataset

Network	EN	SD	MI	SCD	MS-SSIM	$F M I_d c t$	$F M I_w$	$Q_{a b f}$	VIF
NestFuse	7.4418	79.6857	14.8835	1.6504	0.8555	0.3446	0.4395	0.5077	0.9009
NestFuse-HOG	7.4604	80.4892	14.9209	1.6688	0.8597	0.3230	0.4332	0.5062	0.9172
Res2Fusion	7.3300	73.6833	14.6599	1.6476	0.8630	0.3697	0.4326	0.5083	0.7806
Res2Fusion-HOG	7.3469	74.5431	14.6938	1.6526	0.8616	0.3665	0.4318	0.5099	0.7947
UNFusion	7.4137	79.4455	14.8275	1.6355	0.8442	0.3386	0.4361	0.5102	0.9033
UNFusion-HOG	7.4231	80.0186	14.8462	1.6369	0.8416	0.3392	0.4406	0.5119	0.9127

Table 4. Quantitative comparison on the TNO dataset

View table

Table 4. Quantitative comparison on the TNO dataset

Network	EN	SD	MI	SCD	MS-SSIM	$F M I_d c t$	$F M I_w$	$Q_{a b f}$	VIF
NestFuse	6.9198	82.7523	13.8397	1.7335	0.8625	0.3580	0.4371	0.4870	0.7865
NestFuse-grad	6.9091	82.2634	13.8181	1.7390	0.8652	0.3443	0.4337	0.4868	0.7781
NestFuse+grad	6.9297	83.5418	13.8593	1.7433	0.8665	0.3489	0.4347	0.4851	0.7928
NestFuse- perceptual	6.8998	81.2038	13.7995	1.7347	0.8630	0.3500	0.4353	0.4878	0.7684
NestFuse-HOG	6.9435	84.5295	13.8870	1.7491	0.8684	0.3426	0.4323	0.4892	0.7995

Table 5. Quantitative comparison on the RoadScene dataset

View table

Table 5. Quantitative comparison on the RoadScene dataset

Network	EN	SD	MI	SCD	MS-SSIM	$F M I_d c t$	$F M I_w$	$Q_{a b f}$	VIF
NestFuse	7.4418	79.6857	14.8835	1.6504	0.8555	0.3446	0.4395	0.5077	0.9009
NestFuse-grad	7.4475	79.8023	14.8950	1.6603	0.8585	0.3270	0.4353	0.5061	0.9032
NestFuse+grad	7.4394	79.3834	14.8789	1.6566	0.8577	0.3345	0.4366	0.5056	0.8977
NestFuse- perceptual	7.4368	79.3249	14.8735	1.6522	0.8559	0.3346	0.4375	0.5076	0.8947
NestFuse-HOG	7.4604	80.4892	14.9209	1.6688	0.8597	0.3230	0.4332	0.5062	0.9172

Tools

Get Citation

Copy Citation Text

Yuxin Long, Wenjie Lai, Huaiyuan Zhang, Hongbo Zhang, Chengshi Li, Ziji Liu. Soft Histogram of Gradients Loss: A loss Function for Optimization of the Image Fusion Networks[J]. Laser & Optoelectronics Progress, 2023, 60(24): 2411001

Download Citation

EndNote(RIS)BibTex Plain Text

Set citation alerts for article

Save article for my favorites