Journal of Infrared and Millimeter Waves, Volume. 40, Issue 5, 696(2021)

Infrared and visible image fusion based on edge-preserving and attention generative adversarial network

Wen-Qing ZHU1,2,3, Xin-Yi TANG1,3、*, Rui ZHANG1,2,3, Xiao CHEN1,2,3, and Zhuang MIAO1,2,3
Author Affiliations
  • 1Shanghai Institute of Technical Physics,Chinese Academy of Sciences,Shanghai 200083,China
  • 2University of Chinese Academy of Sciences,Beijing 100049,China
  • 3Key Laboratory of Infrared System Detection and Imaging Technology,Chinese Academy of Sciences,Shanghai 200083,China
  • show less
    Figures & Tables(12)
    Architecture of the proposed EAGAN. CA Block:channel attention block,SA Block:spatial attention block,BN:batch normalization,FC:fully connected layer,Conv:corresponding kernel size(k),number of feature maps(n)and stride(s)indicated for each convolutional layer
    Architecture of Attention Block. GAP:Global Average Pooling,GMP:Global Max Pooling,r:scaling factor,Conv:corresponding kernel size(k),number of feature maps(n)and stride(s)indicated for each convolutional layer
    Qualitative comparison of different algorithms on 4 typical infrared and visible image pairs. From top to bottom:visible image,infrared image,fusion results of ASR,GFF,GTF,DenseFuse,FusionGAN,RCGAN and our algorithm.
    Qualitative comparison of different algorithms on 5 typical infrared and visible image pairs from TNO dataset. From left to right:Duine sequence,Nato_camp_sequence,Kaptein_1123,men in front of house and soldier_behind_smoke_3. From top to bottom:visible image,infrared image,fusion results of ASR,GFF,GTF,DenseFuse,FusionGAN,RCGAN and our algorithm.
    Qualitative comparison of different algorithms on 4 typical infrared and visible image pairs from INO dataset. From left to right:ParkingSnow,GroupFight,MultipleDeposit,ClosePerson. From top to bottom:visible image,infrared image,fusion results of ASR,GFF,GTF,DenseFuse,FusionGAN,RCGAN and our algorithm.
    Attention weight maps:(a)the infrared image;(b)the visible image;(c)the fused result of our proposed EAGAN;(d)Output result of the third attention block;(e)Channel Attention weight map;(f)Spatial Attention weight map
    The effect of attention mechanism on fusion results:(a)fusion result of the network without attention mechanism;(b)fusion result of our algorithm.
    Fusion results when the loss function of the generator changes:(a)ℒG=λ1ℒperceptual;(b)ℒG=λ2ℒedge;(c)ℒG=ℒEAGANG;(d)ℒG=ℒEAGANG+λ1ℒperceptual;(e)ℒG=ℒEAGANG+λ2ℒedge;(f)ℒG=λ1ℒperceptual+λ2ℒedge;(g)result of EAGAN.
    • Table 1. Quantitative comparison of different algorithms on RoadScene dataset

      View table
      View in Article

      Table 1. Quantitative comparison of different algorithms on RoadScene dataset

      ASRGFFGTFDenseFuseFusionGANRCGANOurs
      EN6.927.267.267.246.847.147.30
      SCD1.271.230.981.710.861.191.54
      SF13.1113.609.1211.979.359.5015.62
      EI0.220.220.160.200.160.180.27
    • Table 2. Quantitative comparison of different algorithms on TNO dataset

      View table
      View in Article

      Table 2. Quantitative comparison of different algorithms on TNO dataset

      ASRGFFGTFDenseFuseFusionGANRCGANOurs
      EN6.446.846.936.876.356.777.08
      SCD1.611.360.971.791.301.411.67
      SF8.939.558.318.546.597.4111.59
      EI0.130.140.130.140.110.130.19
    • Table 3. Quantitative comparison of different algorithms on INO dataset

      View table
      View in Article

      Table 3. Quantitative comparison of different algorithms on INO dataset

      ASRGFFGTFDenseFuseFusionGANRCGANOurs
      EN6.947.147.027.096.626.977.23
      SCD1.401.291.031.691.021.181.53
      SF16.8017.3314.7214.3412.7113.1219.40
      EI0.250.260.210.220.190.210.30
    • Table 4. Comparison of effects of attention mechanism on fusion results

      View table
      View in Article

      Table 4. Comparison of effects of attention mechanism on fusion results

      ENSCDSFEI
      RoadScene无注意力机制方法7.261.5216.020.27
      本文方法7.301.5415.620.27
      TNO无注意力机制方法6.931.6311.620.19
      本文方法7.081.6711.590.19
    Tools

    Get Citation

    Copy Citation Text

    Wen-Qing ZHU, Xin-Yi TANG, Rui ZHANG, Xiao CHEN, Zhuang MIAO. Infrared and visible image fusion based on edge-preserving and attention generative adversarial network[J]. Journal of Infrared and Millimeter Waves, 2021, 40(5): 696

    Download Citation

    EndNote(RIS)BibTexPlain Text
    Save article for my favorites
    Paper Information

    Category: Research Articles

    Received: Oct. 29, 2020

    Accepted: --

    Published Online: Sep. 29, 2021

    The Author Email: Xin-Yi TANG (gq227@mail.sitp.ac.cn)

    DOI:10.11972/j.issn.1001-9014.2021.05.017

    Topics