Infrared and Visible Image Fusion: Statistical Analysis, Deep Learning Approaches and Future Prospects

Yifei Wu; Rui Yang; Lü Qishen; Yuting Tang; Chengmin Zhang; Shuaihui Liu

doi:10.3788/LOP232360

Category

Name

Meaning

↑∕↓

Formula

Explanation

Information theory

Entropy

EN^［154］，measures the amount of information or information complexity of an image in terms of its gray levels

↑

E N = - \sum_{l = 0}^{L - 1} p_{l} l o g_{2} p_{l}

$L$ is the number of grayscale levels.

$P_{l}$ is the normalized histogram of corresponding grayscale levels in the fused image

Cross entropy

CE^［155］，evaluates the difference between the fused image and the source image

↓

C E = \frac{C E_{A, F} + C E_{B, F}}{2}

C E_{A, F}

，and

C E_{B, F}

are the cross-entropy between images

A

and

B

and the fused image

F

C E_{X, F} = \sum_{i = 0}^{255} h_{X} (i) l o g_{2} \frac{h_{X} (i)}{h_{F} (i)}

h_{X} (i)

is the normalized histogram of image

X

Mutual information

MI^［156］，quantifies the information consistency between the fused image and the source image

↑

M I = M I_{A, F} + M I_{B, F}

$M I_{A, F}$ and $M I_{B, F}$ respectively represent the amount of information transferred from image $A$ and image $B$ to the fused image $F$

$p_{X} (x)$ and $p_{F} (f)$ respectively represent the edge histograms of the source image $X$ and the fused image $F$ . $p_{X, F} (x, f)$ is their joint histogram

M I_{X, F} = \sum_{x, f}^{n} p_{X, F} (x, f) l o g_{10} \frac{p_{X, F} (x, f)}{p_{X} (x) p_{F} (f)}

Feature mutual information

FMI^［157］，focuses on the similarity of image features and information sharing

↑

F M I = M I_{\dot{A}, \dot{F}} + M I_{\dot{B}, \dot{F}}

\overset{\cdot}{A}

，

\overset{\cdot}{B}

and

\overset{\cdot}{F}

respectively represent the feature maps of images

A

，

B

，and the fused image

F

Normalized mutual information

NMI^［158］，quantifies the degree of information sharing between the fused image and the source image

↑

N M I = 2 [\frac{M I_{A, F}}{H (A) + H (F)} + \frac{M I_{B, F}}{H (B) + H (F)}]

M I_{A, F}

and

M I_{B, F}

represent the amount of information transferred from images

A

and

B

to the fused image

F

H (X)

represents the entropy of image

X

Peak

signal-to-noise ratio

PSNR^［159］，measures the degree of distortion between the fused image and the source image

↑

P S N R = 10 l o g_{10} \frac{r^{2}}{M S E}

r

represents the peak value of the fused image

$M S E = \frac{M S E_{A, F} + M S E_{B, F}}{2}$

$M S E_{X, F} = \frac{1}{M N} \sum_{i = 0}^{M - 1} \sum_{j = 0}^{N - 1} {[X (i, j) - F (i, j)]}^{2}$

M S E_{A, F}

and

M S E_{B, F}

represent the mean squared error between the fused image

F

and images

A

，

B

Image

feature

Average gradient

AG^［160］，measures the edge information and image structure of the fused image

↑

A G = \frac{1}{M N} \sum_{i = 1}^{M} \sum_{j = 1}^{N} \sqrt[]{\frac{\nabla F_{x}^{2} (i, j) + \nabla F_{y}^{2} (i, j)}{2}}

\begin{array}{l} \nabla F_{x} (i, j) = \\ F (i, j) - F (i + 1, j) \end{array}

\begin{array}{l} \nabla F_{y} (i, j) = \\ F (i, j) - F (i, j + 1) \end{array}

Edge intensity

EI，evaluates the degree of edge information preservation

↑

E I = \sqrt[]{{\vec{S}}_{x}^{2} + {\vec{S}}_{y}^{2}}

${\vec{S}}_{x} = \vec{F} * {\vec{h}}_{x}$

${\vec{S}}_{y} = \vec{F} * {\vec{h}}_{y}$

${\vec{h}}_{x} = [\begin{matrix} - 1 & 0 & 1 \\ - 2 & 0 & 2 \\ - 1 & 0 & 1 \end{matrix}]$

${\vec{h}}_{y} = [\begin{matrix} - 1 & - 2 & - 1 \\ 0 & 0 & 0 \\ 1 & 2 & 1 \end{matrix}]$

Standard deviation

SD^［161］，measures

the variation or discreteness of pixel values in the image

↑

S D = \sqrt[]{\sum_{i = 1}^{M} \sum_{j = 1}^{N} {[F (i, j) - μ]}^{2}}

$F (i, j)$ represents the value of each pixel point.

$μ$ represents the pixel average value of the fused image

Spatial frequency

SF^［162］，measures the preservation degree of high-frequency and low-frequency information in the fused image

↑

S F = \sqrt[]{R F^{2} + C F^{2}}

$\begin{array}{l} R F = \\ \sqrt[]{\sum_{i = 1}^{M} \sum_{j = 1}^{N} {[F (i, j) - F (i, j - 1)]}^{2}} \end{array}$

$\begin{array}{l} C F = \\ \sqrt[]{{\sum_{i = 1}^{M} \sum_{j = 1}^{N} [F (i, j) - F (i - 1, j)]}^{2}} \end{array}$

Edge based similarity measurement

$Q^{\frac{A B}{F}}$ ^［163］，computes

the amount

of edge information transferred from the source image to the fused image

↑

Q^{\frac{A B}{F}} = \frac{\sum_{i = 1}^{N} \sum_{j = 1}^{M} Q^{A F} (i, j) w^{A} (i, j) + Q^{B F} (i, j) w^{B} (i, j)}{\sum_{i = 1}^{N} \sum_{j = 1}^{M} [w^{A} (i, j) + w^{B} (i, j)]}

$\begin{array}{l} Q^{X F} (i, j) = \\ Q_{g}^{X F} (i, j) Q_{a}^{X F} (i, j) \end{array}$ .

$Q_{g}^{X F} (i, j)$ and $Q_{a}^{X F} (i, j)$ are respectively the edge strength and

orientation-preserving

values at $(i, j)$ .

$w^{A}$ and $w^{B}$ is the importance weight of the source image

Image

structure

Structural similarity index measure

SSIM^［164］，reflects the degree of structural similarity between images

↑

\begin{array}{l} S S I M_{X, F} = \\ \sum_{x, f}^{n} \frac{2 μ_{x} μ_{f} + C_{1}}{μ_{x}^{2} + μ_{f}^{2} + C_{1}} \cdot \frac{2 σ_{x} σ_{f} + C_{2}}{σ_{x}^{2} + σ_{f}^{2} + C_{2}} \cdot \frac{σ_{x f} + C_{3}}{σ_{x} σ_{f} + C_{3}} \end{array}

$x$ and $ƒ$ respectively represent image blocks of the source image and the fused image in the sliding window

$μ_{x}$ and $μ_{f}$ is the pixel average value.

$σ_{x}$ and $σ_{f}$ is the standard deviation. $σ_{x f}$ is the covariance between the source image and the fused image.

$C_{1}$ ， $C_{2}$ ， $C_{3}$ is the parameter of the stable algorithm.

$S S I M$ is the overall structural similarity index

S S I M = S S I M_{A, F} + S S I M_{B, F}

Yang’s

metric

Q_{Y}

^［165］，calculates the amount of preserved structural information

↑

Q_{Y} = \{\begin{array}{l} \begin{array}{l} λ (w) S S I M (A, F |w) + [1 - λ (w)] S S I M \\ (B, F |w), S S I M (A, B |w) \geq 0.75, \end{array} \\ \begin{array}{l} m a x [S S I M (A, F |w), S S I M (B, F |w)], \\ S S I M (A, B |w) < 0.75 \end{array} \end{array}

ω

represents the local window

λ (w) = \frac{s (A w)}{s (A w) + s (B w)}

Visual perception

Chen-Blum metric

Q_{C B}

^［166］，measures the similarity of primary visual features of human vision

↑

\begin{array}{l} Q_{C B} = \\ \frac{1}{M N} [\sum_{i = 1}^{N} \sum_{j = 1}^{M} β_{A} (i, j) W_{A, F} (i, j) + β_{B} (i, j) W_{B, F} (i, j)] \end{array}

$W_{A, F} (i, j)$ and $W_{B, F} (i, j)$ are the contrasts transferred from source images $A$ and $B$ to the fused image $F$ .

$β_{A}$ and $β_{B}$ is their saliency map

Visual information fidelity for fusion

VIFF^［167］，evaluates the fidelity of visual information fusion

↑

\begin{matrix} V I F F = \sum_{k = 1}^{N} p_{i} \times V I F F_{k} (Z, K, H) \end{matrix}

F V I N D

and

F V I D

respectively represent non-distorted visual information similarity and distorted visual information similarity.

p_{i}

is the weighted coefficient of the

i

-th sub-band

\begin{matrix} V I F F_{k} (Z, K, H) = \frac{\sum_{b}^{n} F V I D_{k, b} (Z, K, H)}{\sum_{b}^{n} F V I N D_{k, b} (Z, K, H)} \end{matrix}

Correlation

Correlation coefficient

CC^［1］，measures the degree of linear correlation between the fused image and the source image

↑

C C = \frac{r_{A F} + r_{B F}}{2}

$\bar{X}$ represents the mean of the source image $X$ .

$F (i, j)$ represents the value of

each pixel point.

$μ$ represents the pixel average value of the fused image $F$

r_{X F} = \frac{\sum_{i = 1}^{M} \sum_{j = 1}^{N} [X (i, j) - \bar{X}] [F (i, j) - μ]}{\sqrt[]{\sum_{i = 1}^{M} \sum_{j = 1}^{N} {[X (i, j) - \bar{X}]}^{2} \{\sqrt[]{\sum_{i = 1}^{M} \sum_{j = 1}^{N} {[F (i, j) - μ]}^{2}}\}}}

Nonlinear correlation coefficient

NCC^［168］，measures the degree of nonlinear correlation between the fused image and the source image

↑

N C C = \frac{N C C_{A F} + N C C_{B F}}{2}

$S_{i}$ is the quantity of samples in the $i$ -th rank of the distribution

$b$ represents the total number of ranks. $S$ represents the total number of sample pairs

N C C_{X F} = 2 + \sum_{i = 1}^{b^{2}} \frac{S_{i}}{S} l o g_{b} \frac{S_{i}}{S}

Yifei Wu, Rui Yang, Lü Qishen, Yuting Tang, Chengmin Zhang, Shuaihui Liu. Infrared and Visible Image Fusion: Statistical Analysis, Deep Learning Approaches and Future Prospects[J]. Laser & Optoelectronics Progress, 2024, 61(14): 1400004

Category: Reviews

Received: Oct. 24, 2023

Accepted: Dec. 25, 2023

Published Online: Jul. 25, 2024

The Author Email: Rui Yang (yangrui@jou.edu.cn)

DOI:10.3788/LOP232360

CSTR:32186.14.LOP232360

Table 1. Literature retrieval and statistics for IVIF

Table 1. Literature retrieval and statistics for IVIF

Table 2. Some traditional IVIF methods

Table 2. Some traditional IVIF methods

Table 3. AE-based IVIF methods

Table 3. AE-based IVIF methods

Table 4. CNN-based IVIF methods

Table 4. CNN-based IVIF methods

Table 5. GAN-based IVIF methods

Table 5. GAN-based IVIF methods

Table 6. Transformer based IVIF methods

Table 6. Transformer based IVIF methods

Table 7. Objective evaluation metrics

Table 7. Objective evaluation metrics