Multimodal Image Fusion Based on Generative Adversarial Networks

Fig. 10. Effect of λ on objective evaluation index of fused image. (a) The first set of fused images; (b) the second set of fused images; (c) the third set of fused images

Download full size

Fig. 11. Image fusion results. (a) Longwave infrared; (b) shortwave infrared; (c) visible light; (d) DTCWT_SR; (e) NSST_NSCT; (f) CNN; (g) CSR; (h) proposed method

Download full size

Table 1. Parameters of generator
View table
Table 1. Parameters of generator
Layer Filter size /step Output size
Conv1 3×3 /1 128×128×64
Res(7 units) 3×3 /1 128×128×64
3×3 /1 128×128×64
Conv9 3×3 /1 128×128×256
Conv10 3×3 /1 128×128×1

Table 2. Parameters of discriminator

View table

Table 2. Parameters of discriminator

Layer	Filter size/step	Output size
Conv1	3×3 /1	128×128×64
Conv2	3×3 /2	64×64×128
Conv3	3×3 /2	32×32×256
Conv4	3×3 /2	16×16×512
Conv5	3×3 /1	16×16×256
Conv6	1×1 /1	16×16×128
Res	1×1 /1	16×16×64
	3×3 /1	16×16×64
	3×3 /1	16×16×128
Fc	-	1

Table 3. Label image selection table
View table
Table 3. Label image selection table
Fusion method SD AG Con CC IE MI VIFF
LP 49.019 4.211 40.812 0.424 7.223 5.608 0.466
DWT 44.772 3.826 36.650 0.342 7.175 5.415 0.442
NSCT 41.123 4.061 34.816 0.437 6.953 5.222 0.469
NSST 40.904 4.149 34.902 0.441 6.932 5.201 0.467

Table 4. Comparison of evaluation index of fusion results

View table

Table 4. Comparison of evaluation index of fusion results

Image	Fusion method	SD	AG	Con	CC	IE	MI	VIFF
No. 1	DTCWT_SR	35.471	3.150	23.916	0.409	6.968	4.869	0.505
	NSCT_NSST	19.621	3.163	11.911	0.406	6.147	2.853	0.522
	CNN	34.928	2.885	23.254	0.408	6.961	4.610	0.363
	CSR	12.081	1.953	7.570	0.418	5.534	2.663	0.359
	Proposed method	38.011	8.129	27.793	0.431	6.977	2.552	0.301
No. 2	DTCWT_SR	26.274	7.245	25.927	0.143	5.984	1.492	0.367
	NSCT_NSST	23.268	7.270	14.815	0.306	6.026	1.542	0.376
	CNN	29.696	4.985	22.754	0.016	2.373	1.312	0.211
	CSR	25.271	5.543	15.989	0.322	6.032	2.540	0.322
	Proposed method	25.527	4.374	13.731	0.381	6.057	2.567	0.461
No. 3	DTCWT_SR	21.854	5.187	15.038	0.427	6.475	1.256	0.415
	NSCT_NSST	22.692	5.370	15.7166	0.471	6.806	1.542	0.419
	CNN	38.590	4.961	25.609	0.441	6.910	2.799	0.392
	CSR	23.054	3.868	15.726	0.499	6.450	2.391	0.382
	Proposed method	41.089	4.310	30.929	0.593	6.938	3.093	0.420
No. 4	DTCWT_SR	47.492	3.545	23.176	0.422	6.315	2.773	0.298
	NSCT_NSST	35.373	3.605	15.136	0.445	6.831	2.531	0.317
	CNN	54.597	3.163	29.778	0.436	6.283	2.862	0.351
	CSR	38.587	2.217	17.656	0.452	6.796	3.331	0.375
	Proposed method	32.400	8.452	22.767	0.457	6.938	2.055	0.384
No. 5	DTCWT_SR	40.006	5.058	32.554	0.025	7.308	2.976	0.407
	NSCT_NSST	24.886	5.147	18.059	0.489	6.615	1.812	0.428
	CNN	40.559	3.951	34.331	0.271	7.316	2.133	0.346
	CSR	26.239	2.526	20.707	0.515	6.642	3.007	0.277
	Proposed method	40.677	5.974	33.890	0.569	7.321	2.106	0.473
No. 6	DTCWT_SR	54.458	3.038	44.020	0.042	7.744	2.986	0.524
	NSCT_NSST	25.990	3.047	18.728	0.455	6.686	2.254	0.552
	CNN	45.549	2.538	35.269	0.193	7.333	1.839	0.208
	CSR	28.803	1.523	21.125	0.429	6.691	2.858	0.160
	Proposed method	46.425	5.643	36.462	0.476	7.453	2.993	0.648
No. 7	DTCWT_SR	54.848	3.555	36.984	0.044	7.420	3.485	0.484
	NSCT_NSST	27.527	3.550	21.490	0.460	6.813	1.807	0.504
	CNN	45.880	2.945	37.129	0.334	7.104	2.627	0.440
	CSR	27.853	1.834	21.168	0.472	6.701	2.968	0.326
	Proposed method	47.024	4.259	37.523	0.500	7.487	2.461	0.606
No. 8	DTCWT_SR	55.890	2.940	47.062	0.771	4.357	3.612	0.169
	NSCT_NSST	52.610	4.872	47.647	0.868	4.832	3.906	0.380
	CNN	87.236	6.208	73.907	0.721	5.688	3.933	0.294
	CSR	54.208	4.753	47.214	0.794	4.245	3.383	0.285
	Proposed method	92.152	15.279	76.870	0.943	4.333	4.947	0.505

Tools

Get Citation

Copy Citation Text

Xiaoli Yang, Suzhen Lin, Xiaofei Lu, Lifang Wang, Dawei Li, Bin Wang. Multimodal Image Fusion Based on Generative Adversarial Networks[J]. Laser & Optoelectronics Progress, 2019, 56(16): 161004

Download Citation

EndNote(RIS)BibTex Plain Text

Set citation alerts for article

Save article for my favorites