Visible-to-infrared Image Translation Based on an Improved Conditional Generative Adversarial Nets

Layers name	Output size	Networks layers
Down1	64×64	4×4Conv2d，96，Stride=4
Stages1	64×64	$[\begin{array}{l} 7 \times 7,96 \\ 7 \times 7,96 \end{array}] \times 3$
Down2	32×32	2×2Conv2d，192，Stride=2
Stages2	32×32	$[\begin{array}{l} 7 \times 7,192 \\ 7 \times 7,192 \end{array}] \times 3$
Down3	16×16	2×2Conv2d，384，Stride=2
Stages3	16×16	$[\begin{array}{l} 7 \times 7,384 \\ 7 \times 7,384 \end{array}] \times 9$
Down4	8×8	2×2Conv2d，768，Stride=2
Stages4	8×8	$[\begin{array}{l} 7 \times 7,768 \\ 7 \times 7,768 \end{array}] \times 3$
Dconv1	16×16	$[\begin{matrix} 1 \times 1,768, C o n v 2 d \\ 3 \times 3,384, C o n v T r a n s p o s e 2 d \\ 1 \times 1,384, C o n v 2 d \end{matrix}]$
Dconv2	32×32	$[\begin{matrix} 1 \times 1,768, C o n v 2 d \\ 3 \times 3,192, C o n v T r a n s p o s e 2 d \\ 1 \times 1,192, C o n v 2 d \end{matrix}]$
Dconv3	64×64	$[\begin{matrix} 1 \times 1,384, C o n v 2 d \\ 3 \times 3,96, C o n v T r a n s p o s e 2 d \\ 1 \times 1,96, C o n v 2 d \end{matrix}]$
Dconv4	128×128	$[\begin{matrix} 1 \times 1,192, C o n v 2 d \\ 3 \times 3,48, C o n v T r a n s p o s e 2 d \\ 1 \times 1,48, C o n v 2 d \end{matrix}]$
Final Dconv	256×256	$[\begin{matrix} 4 \times 4,24, C o n v T r a n s p o s e 2 d \\ 3 \times 3,12, C o n v 2 d \\ 3 \times 3,3, C o n v 2 d \end{matrix}]$

Table 2. Adversarial networks structure parameter

View table

View in Article

Table 2. Adversarial networks structure parameter

Layers name	Output size	Networks layers
Down	256×256，128×128，64×64	$2 \times 2 A v g P o o l, S t r i d e = 2$
Conv1	128×128，64×64，32×32	$4 \times 4 C o n v 2 d, 64, S t r i d e = 2$
Conv2	64×64，32×32，16×16	$4 \times 4 C o n v 2 d, 128, S t r i d e = 2$
Conv3	32×32，16×16，8×8	$4 \times 4 C o n v 2 d, 256, S t r i d e = 2$
Conv4	32×32，16×16，8×8	$1 \times 1 C o n v 2 d, 512, S t r i d e = 1$
Conv5	32×32，16×16，8×8	$1 \times 1 C o n v 2 d, 1 024, S t r i d e = 1$

Table 3. The objective comparison results of ablation experiment

View table

View in Article

Table 3. The objective comparison results of ablation experiment

Evaluation indexes	SSIM↑	MS-SSIM↑	PSNR↑	LPIPS↓	FID↓	NCC↑
Pix2Pix	0.910 3	0.799 4	26.979 9	0.085 6	51.231 1	0.931 1
Baseline+SPatchGAN	0.918 9	0.834 6	27.247 8	0.077 0	48.790 9	0.937 0
Baseline+Upsampling1	0.920 2	0.853 4	30.086 3	0.057 9	43.034 6	0.941 4
Baseline+Upsampling2	0.928 0	0.855 0	29.937 5	0.055 3	42.220 2	0.942 9
Baseline+Upsampling3	0.943 9	0.859 5	30.580 4	0.050 7	38.969 9	0.948 9
Baseline+Upsampling4	0.945 4	0.856 6	30.620 7	0.050 3	38.895 5	0.948 7

Table 4. The objective experiment results comparison of different algorithms

View table

View in Article

Table 4. The objective experiment results comparison of different algorithms

VEDAI	SSIM↑	MS-SSIM↑	PSNR↑	LPIPS↓	FID↓	NCC↑
Pix2Pix	0.910 3	0.799 4	26.979 9	0.085 6	51.231 1	0.931 1
ThermalGAN	0.946 6	0.853 9	30.048 9	0.052 5	39.967 8	0.947 3
I-GANs	0.814 6	0.744 2	27.287 7	0.158 1	119.674 3	0.893 5
InfraGAN	0.948 7	0.858 5	29.987 7	0.051 1	39.796 1	0.948 0
Ours	0.955 1	0.881 6	31.329 4	0.042 3	33.954 0	0.960 4
Ours（512×512）	0.950 8	0.854 7	31.223 7	0.059 7	37.853 0	0.960 1
Ours（1024×1024）	0.955 5	0.848 1	31.864 9	0.085 6	53.615 7	0.951 0
OSU	SSIM↑	MS-SSIM↑	PSNR↑	LPIPS↓	FID↓	NCC↑
Pix2Pix	0.901 5	0.885 4	24.187 6	0.118 8	84.363 2	0.975 6
ThermalGAN	0.904 3	0.906 1	29.239 8	0.132 2	78.079 8	0.974 5
I-GANs	0.881 2	0.867 7	27.767 0	0.160 3	107.761 4	0.964 3
InfraGAN	0.905 1	0.904 6	29.025 5	0.137 4	89.047 5	0.973 1
Ours	0.923 4	0.936 8	31.331 2	0.064 5	57.973 5	0.983 9
KAIST	SSIM↑	MS-SSIM↑	PSNR↑	LPIPS↓	FID↓	NCC↑
Pix2Pix	0.827 7	0.593 1	22.732 8	0.201 7	77.878 9	0.922 6
ThermalGAN	0.854 1	0.613 8	23.550 4	0.283 1	112.654 8	0.937 1
I-GANs	0.844 1	0.572 0	22.950 3	0.361 8	156.585 2	0.927 9
InfraGAN	0.844 5	0.608 2	23.021 4	0.181 0	63.930 8	0.929 5
Ours	0.869 2	0.700 8	24.452 4	0.112 3	27.533 1	0.948 3

Table 5. Network structure of different algorithms
View table
View in Article
Table 5. Network structure of different algorithms
Method Generator Discriminator Loss
L1 loss SSIM loss
Pix2Pix ResNet9block PatchGAN √
ThermalGAN UNet PatchGAN √
I-GANs D-Linket34 PatchGAN √
InfraGAN UNet UNetGAN √ √
Ours UConvNext SPatchGAN √

Table 6. EPE of six matching algorithms

View table

View in Article

Table 6. EPE of six matching algorithms

EPE VEDAI	SIFT	SURF	ORB	D2-Net	SuperGlue	LoFTR
RGB	0.131 7	2.334 4	8.151 8	0.767 3	0.334 8	0.136 5
Pix2Pix	0.311 6	0.245 3	0.660 1	0.387 1	0.477 0	0.052 6
ThermalGAN	0.118 0	0.170 6	0.196 7	0.284 3	0.302 3	0.027 2
I-GANs	0.631 0	0.795 4	1.029 8	0.542 0	0.433 1	0.070 4
InfraGAN	0.109 6	0.166 2	0.287 3	0.265 9	0.314 5	0.021 6
Ours	0.081 8	0.154 7	0.106 5	0.238 3	0.257 1	0.017 8
EPE OSU	SIFT	SURF	ORB	D2-Net	SuperGlue	LoFTR
RGB	52.614 8	28.740 6	38.851 5	1.695 6	0.877 6	2.533 4
Pix2Pix	0.105 3	0.099 9	0.125 5	0.218 2	0.178 0	0.043 3
ThermalGAN	0.126 1	0.105 0	0.138 0	0.233 4	0.173 5	0.042 7
I-GANs	0.147 1	0.117 4	0.154 0	0.259 3	0.212 5	0.058 7
InfraGAN	0.122 7	0.105 6	0.140 0	0.224 6	0.163 8	0.040 0
Ours	0.080 8	0.078 9	0.088 8	0.167 1	0.145 6	0.034 9
EPE KAIST	SIFT	SURF	ORB	D2-Net	SuperGlue	LoFTR
RGB	40.716 0	36.108 2	33.319 1	2.765 0	4.387 7	10.397 7
Pix2Pix	2.885 9	1.744 7	2.430 1	0.855 9	1.045 2	0.546 8
ThermalGAN	1.498 3	1.533 5	1.055 8	0.970 2	0.944 1	0.807 6
I-GANs	2.089 5	2.399 4	1.814 8	1.201 2	1.523 7	2.574 3
InfraGAN	2.024 1	1.438 6	2.184 2	0.726 6	0.803 6	0.385 4
Ours	0.630 3	0.552 1	0.622 2	0.477 9	0.495 5	0.181 3

Tools

Get Citation

Copy Citation Text

Decao MA, Yong XIAN, Juan SU, Shaopeng LI, Bing LI. Visible-to-infrared Image Translation Based on an Improved Conditional Generative Adversarial Nets[J]. Acta Photonica Sinica, 2023, 52(4): 0410003

Download Citation

EndNote(RIS)BibTex Plain Text

Set citation alerts for article

Save article for my favorites