Acta Optica Sinica, Volume. 45, Issue 8, 0810001(2025)

Dual‑Branch Multimodal Medical Image Fusion Based on Local and Global Information Collaboration

Yu Shen1... Jiaying Liu1,*, Jiarong Yan1, Ruoxuan Wang1, Yukun Ma1, Jiangcheng Li1, Shan Bai1, Ziyi Wei1,2, Yangyang Li1 and Zhenkai Qiang1 |Show fewer author(s)
Author Affiliations
  • 1School of Electronics and Information Engineering, Lanzhou Jiaotong University, Lanzhou 730070, Gansu, China
  • 2Gansu Cuiying Information Technology Co., Ltd., Lanzhou 730030, Gansu, China
  • show less
    Figures & Tables(18)
    Overall framework of this model
    Multi-scale feature extraction module
    Deep local feature enhancement module
    Multi-dimensional joint local fusion unit
    Global fusion module based on cross-attention mechanism
    Comparison of MRI-PET visualization fusion results of “mild Alzheimer disease”. (a) MRI; (b) PET; (c) EMFusion; (d) FATFusion; (e) IFCNN; (f) MATR; (g) CNP; (h) INet; (i) NSST; (j) EMMA; (k) PIAFusion; (l) ours
    Comparison of MRI-CT visualization fusion results of “hypertensive encephalopathy”. (a) CT; (b) MRI; (c) EMFusion; (d) FATFusion; (e) IFCNN; (f) MATR; (g) CNP; (h) INet; (i) NSST; (j) EMMA; (k) PIAFusion; (l) ours
    Comparison of MRI-SPECT visualization fusion results of “hypertensive encephalopathy”. (a) MRI; (b) SPECT; (c) EMFusion; (d) FATFusion; (e) IFCNN; (f) MATR; (g) CNP; (h) INet; (i) NSST; (j) EMMA; (k) PIAFusion; (l) ours
    Fusion results of “MRI-SPECT” images with different loss functions
    Quantitative analysis of parameters. (a) Loss parameter α; (b) loss parameter β; (c) loss parameters γ; (d) channel partitioning parameter c
    Qualitative comparison of MRI-CT ablation experiments
    Qualitative comparison of MRI-PET ablation experiments
    Radar charts of MRI-PET and MRI-CT of ablation experiments. (a) MRI-PET; (b) MRI-CT
    • Table 1. Experimental environment parameters

      View table

      Table 1. Experimental environment parameters

      EnvironmentConfiguration
      GPUNVIDIA RTX 3090
      CPUIntel Core i5-13600KF
      Memory32 GB
      Software environmentPython 3.8.10
      PyTorch1.8.1
      CUDA 11.1.0
      OpenCV 4.5.3
    • Table 2. Objective evaluation results of MRI-PET image fusion for “mild Alzheimer disease”

      View table

      Table 2. Objective evaluation results of MRI-PET image fusion for “mild Alzheimer disease”

      MethodMIENSDSFAGSSIMCC
      EMFusion1.78584.071753.710926.47496.78400.75860.7679
      FATFusion2.04543.958062.242432.75476.62120.74840.7034
      IFCNN1.72283.859663.072329.76927.05040.75390.7932
      MATR1.06624.545045.535223.93056.27820.05060.3462
      CNP1.11334.705060.044128.72167.11270.45630.4276
      INet1.32683.917264.468732.91326.38710.45330.5877
      NSST1.12694.351961.252328.88557.15490.48850.4302
      EMMA1.37234.667962.364821.50187.18990.65590.6654
      PIAFusion1.82943.599851.351923.23446.22120.73190.6783
      Ours1.90274.781263.178826.40957.22590.25310.8012
    • Table 3. Objective evaluation results of “normal aging” MRI-SPECT image fusion

      View table

      Table 3. Objective evaluation results of “normal aging” MRI-SPECT image fusion

      MethodMIENSDSFAGSSIMCC
      EMFusion1.77143.635643.815018.03324.52760.80780.8821
      FATFusion2.24623.650552.816720.16585.19220.79570.8729
      IFCNN1.67683.572047.739317.95034.57800.81370.9136
      MATR1.07383.944237.964614.71563.84560.06990.6927
      CNP1.43674.327849.590115.43264.15790.32800.8656
      INet1.80883.653856.804221.09794.90880.45870.9137
      NSST1.11944.191458.198319.12344.92840.51370.6812
      EMMA1.73534.256256.369314.16883.67230.59630.9125
      PIAFusion1.18123.656860.392215.08663.71590.55110.7191
      Ours1.81484.530747.127716.91215.23200.25160.9182
    • Table 4. Objective evaluation results of MRI-CT image fusion for “hypertensive encephalopathy”

      View table

      Table 4. Objective evaluation results of MRI-CT image fusion for “hypertensive encephalopathy”

      MethodMIENSDSFAGSSIMCC
      EMFusion1.85974.415169.081819.15944.76430.76100.8143
      FATFusion1.99364.063863.934835.31197.94830.67490.8055
      IFCNN1.88464.283866.362934.34657.55650.76920.8132
      MATR1.29644.568453.179924.10686.35150.08480.5648
      CNP1.35894.827168.429425.72646.92640.39850.6538
      INet1.71644.532969.163932.22537.57460.55160.7810
      NSST1.12564.598662.375122.64235.64360.46170.6864
      EMMA1.91895.035568.926424.17736.57850.34730.8246
      PIAFusion1.06234.167365.436426.17955.32610.72630.7458
      Ours1.97725.193669.819940.13599.28410.28910.8338
    • Table 5. Objective evaluation indexes of ablation experiments

      View table

      Table 5. Objective evaluation indexes of ablation experiments

      ImageExperimentDLEMMDFUCGFUENSDSFAGMICC
      MRI-PET1×4.781266.953226.41006.90281.37840.6474
      2×4.049150.315617.21704.94001.23100.6251
      3×4.023163.178824.49746.21081.36020.6873
      4×××3.021268.680117.38693.95771.33150.6517
      54.905681.272326.74988.03861.90270.7974
      MRI-CT1×4.914590.837530.88276.90281.69110.5779
      2×4.967380.898436.88785.75801.69100.6388
      3×4.261780.818427.78925.14251.67400.6531
      4×××3.998778.406423.04927.27561.63290.7154
      55.1993105.43539.71078.62431.70430.8187
    Tools

    Get Citation

    Copy Citation Text

    Yu Shen, Jiaying Liu, Jiarong Yan, Ruoxuan Wang, Yukun Ma, Jiangcheng Li, Shan Bai, Ziyi Wei, Yangyang Li, Zhenkai Qiang. Dual‑Branch Multimodal Medical Image Fusion Based on Local and Global Information Collaboration[J]. Acta Optica Sinica, 2025, 45(8): 0810001

    Download Citation

    EndNote(RIS)BibTexPlain Text
    Save article for my favorites
    Paper Information

    Category: Image Processing

    Received: Jan. 6, 2025

    Accepted: Feb. 18, 2025

    Published Online: Apr. 27, 2025

    The Author Email: Jiaying Liu (1640264144@qq.com)

    DOI:10.3788/AOS250443

    CSTR:32393.14.AOS250443

    Topics