High-resolution non-line-of-sight imaging via polarization differential correlography

Lingfeng Liu; Shuo Zhu; Yi Wei; Jingye Miao; Wenjun Zhang; Lianfa Bai; Enlai Guo; Jing Han

doi:10.3788/COL202523.081104

1. Introduction

Non-line-of-sight (NLOS) imaging uses relay walls to detect, identify, and reconstruct hidden objects around corners. It has applications in key areas such as autonomous driving, remote sensing, and medical diagnosis. However, since the hidden object signal can only be collected after at least three diffuse reflections through the relay wall, and Fourier amplitudes are not sampled in full, the reconstruction of the hidden object is an ill-posed inverse problem^[1].

Existing NLOS imaging methods are categorized into transient and steady-state types, depending on the temporal resolution of the hardware used^[2]. Transient NLOS imaging utilizes high-frequency pulsed lasers and high-temporal-resolution detectors to encode three-dimensional information about hidden objects. It reconstructs hidden objects by calculating the flight time of multiple scattered photons^[3–8]. Transient methods face challenges such as extensive computational requirements and high hardware costs.

Steady-state NLOS imaging has developed reconstruction methods based on speckle correlation^[9–11], computational periscopy^[12], wavefront modulation^[13], edge cameras^[14], and neural networks^[15,16]. These methods need further refinement to address challenges such as a limited imaging field of view, time-consuming sampling iterations, and generalization across application scenarios.

Indirect imaging correlography (IIC) has emerged as a promising approach to enhance the performance of steady-state NLOS imaging^[17–19]. This technique leverages the intensity correlation of multiple speckles in spatial mode on the wall to image hidden objects effectively. To further refine the process, researchers have established a noise model for IIC and developed a deep inverse correlagraphy convolutional neural network^[20]. This approach can reconstruct the hidden object using two short exposure images of 0.125 s. However, it should be noted that this method requires mechanical scanning of the wall with the laser source to achieve independent speckle illumination. This scanning process introduces instability into the optical path of imaging. To address the defect, researchers have explored the chromato-axial difference of speckles, enabling single-shot steady-state NLOS imaging^[21]. Employing a single-shot acquisition method and utilizing uncorrelated speckles on the chromato-axes provides redundant information across multi-spectral channels for the hidden object. Consequently, the method exhibits robustness against system vibrations and color perturbations. However, the challenge arises from mixed illumination originating from multiple light sources, which introduces instability to the power spectrum of hidden objects across different channels.

In this Letter, we propose a high-resolution NLOS imaging method via polarization differential correlography (PDC-NLOS). This approach is predicated on correlography with independent polarized speckles illumination to encode and reconstruct the hidden object. By utilizing the polarization difference of the speckles, the hidden object can be accurately restored. The single-shot polarized speckle illumination strategy avoids mechanical scanning, which reduces acquisition time and provides inherent redundancy against perturbations. With a single source used for indirect illumination, the PDC-NLOS imaging system can provide stable imaging quality of the hidden object. Imaging horizontally and vertically striped objects, the PDC-NLOS imaging method is proven to have millimeter-level imaging resolution.

2. Principle of PDC-NLOS

The imaging process of the proposed PDC-NLOS is shown in Fig. 1. The laser source is expanded by the optical path and illuminated on the relay wall as a virtual source (VS). The coherent beam passes through the half-wave plate, and diffuses through the relay wall, resulting in an initial polarization angle $θ_{pol}$ and a random phase difference. It is assumed that the relay wall behaves as a Lambertian surface with no polarization-dependent bidirectional reflectance distribution function (BRDF), and the hidden object’s albedo is polarization-independent. The output light field of the VS is a polarized laser speckle, which can be described as $E_{V S} (x, y) = (E_{x} \overset{⇀}{x} + E_{y} \overset{⇀}{y}) \exp [j Ψ_{VS} (x, y)] = \sum_{θ_{n} = 0}^{2 π} E_{{VS}_{P}} (x, y; θ_{n}),$ (1)where $E_{x}$ and $E_{y}$ represent the amplitudes of the polarization components of the light field on the horizontal and vertical unit vectors, $\overset{⇀}{x}$ and $\overset{⇀}{y}$ ; $\tan θ_{pol} = \frac{E_{y}}{E_{x}}$ . $Ψ_{VS}$ is a random phase factor generated by the diffusion of VS. $E_{{VS}_{P}}$ is the polarized light field observed based on the polarization angle $θ_{n}$ , and the specific expression can be found in Eq. (S10) in Supplement 1.

Figure 1.Schematic of the PDC-NLOS. (a) Imaging process of the PDC-NLOS. (b) Principal diagram of the PDC-NLOS. LP, linear polarization.

Download full size

View all figures

$E_{V S}$ is transmitted to the surface of the hidden object through free space and is reflected and diffused to generate the emergent light field $E_{O}$ : $E_{O} (x, y) = r_{O} (x, y) \cdot \frac{e^{j k z}}{j λ z} \cdot \sum_{θ_{n} = 0}^{2 π} \iint_{\infty} E_{V S_{p}} (x, y; θ_{n}) \exp [\frac{j π}{λ z} {(z - z_{VS})}^{2}] d x d y,$ (2)where $z$ is the transmission distance of the light field, $z_{VS}$ is the axial position of the VS, and $r_{O}$ is the albedo function of the hidden object.

$E_{O}$ continues to propagate to another part of the relay wall as a virtual detector (VD). When the size and position of the object, as well as the area of the scattering region on the wall, are set within an appropriate range (see Supplement 1), the propagation process from the hidden object to the VD can be regarded as far-field propagation. So the light field $E_{V D}$ at VD can be expressed as $E_{V D} (x, y) = c_{1} F [E_{O} (x, y)] = c_{1} \sum_{θ_{n} = 0}^{2 π} E_{O_{p}} (x, y; θ_{n}) .$ (3)

The speckle intensity $I_{{VD}_{p}} (x, y; θ_{n})$ of the channel corresponding to polarization angle $θ_{n}$ captured by the polarization camera can be expressed as $I_{{VD}_{p}} (x, y; θ_{n}) = c_{2} {| F [E_{O_{p}} (x, y; θ_{n})] |}^{2} + η,$ (4)where in the above two equations, $c_{1}$ and $c_{2}$ are the constant terms, and $η$ is the noise caused by the wall background.

According to the two-point intensity correlation technique and the coherent representation of the polarization speckle intensity^[22], the correlation coefficient of the speckles at the observation angles $θ_{1}$ and $θ_{2}$ can be expressed as $C (θ_{1}, θ_{2}) = \frac{⟨ Δ I (θ_{1}) Δ I (θ_{2}) ⟩}{σ (θ_{1}) σ (θ_{2})} = \frac{\cos θ_{1} \cos θ_{2} + ζ \sin θ_{1} \sin θ_{2}}{(\cos^{2} θ_{1} + ζ \sin^{2} θ_{1}) (\cos^{2} θ_{2} + ζ \sin^{2} θ_{2})},$ (5)where $ζ = \frac{⟨ I_{y} ⟩}{⟨ I_{x} ⟩}$ and $σ$ is the standard deviation of intensity. Letting $ς = \frac{{(1 - ζ)}^{2}}{(1 + ζ^{2})}$ , Eq. (5) can be simplified to $C (θ_{1}, θ_{2}) = (1 - ς) \cos^{2} (θ_{1} - θ_{2}) + ς .$ (6)

As shown in Eq. (6), the correlation coefficient between polarized speckles changes in a sinusoidal fashion with the difference of the polarization angles. Shown in Fig. 2, letting 0.5 be used as the empirical threshold^[23], an appropriate difference of the polarization angles can obtain completely uncorrelated speckles, which can be used to encode the hidden object.

Figure 2.Curve of the polarization angles to the coherent coefficient between polarized speckles.

Download full size

View all figures

Replacing the number of periodograms of the speckle images in deep-inverse correlography^[20] with the number of polarization differential speckles, the power spectrum measurement of $I_{VD}$ can be expressed as $\lim_{N \to \infty} \frac{1}{N} \sum_{n = 1}^{N} {| F^{- 1} [I (x, y; θ_{n})] |}^{2} = r_{O} ⋆ r_{O} (x, y) + δ (Δ x, Δ y) {[\iint_{\infty} r_{O} (x, y) d x d y]}^{2},$ (7)and $\lim_{N \to \infty} {| \frac{1}{N} \sum_{n = 1}^{N} F^{- 1} [I (x, y; θ_{n})] |}^{2} = δ (Δ x, Δ y) {[\iint_{\infty} r_{O} (x, y) d x d y]}^{2},$ (8)where $N$ denotes the number of polarization differential speckles, and $θ_{n}$ denotes the different polarization angle. Letting Eq. (7) subtract Eq. (8), a preliminary autocorrelation of the hidden object can be estimated.

The speckle intensity $I_{VD}$ collected by the camera is proportional to the incident speckle intensity $I_{VS}$ : $I_{VD} (θ_{n}) \propto I_{VS} (θ_{n}) = \frac{1}{2} (J_{x x} + J_{y y}) + R \cos (2 θ_{n} - α),$ (9)where $R = \frac{1}{2} \sqrt{{(J_{x x} - J_{y y})}^{2} + 4 (J_{x y} J_{y x} \cos^{2} β)}$ , and $\tan α = \frac{2 | J_{x y} | \cos β}{J_{x x} - J_{y y}}$ . Obviously, $I_{VD}$ follows the sinusoidal distribution of the polarization angle $θ_{n}$ . It is necessary to correct the intensity values of the collected speckles according to Eq. (9). Moreover, due to the polarization filtering, the average irradiance received by a pixel in the PDC-NLOS is reduced by approximately a factor of two when compared to traditional IIC.

3. System Setup

We built the system as shown in Fig. 3 to verify the imaging capability of the proposed method.

Figure 3.Experimental setup of the PDC-NLOS. (a) System setup diagram; (b) polarized camera and lens used; (c) DMD used to display objects (light blocking stickers are used in practical experiments to reduce interference due to the front surface of DMD).

Download full size

View all figures

The wavelength of the laser is set to 532 nm. The distance from the object to the VS is 300 mm. The distance from the camera to the VD is 500 mm. The hidden object is displayed on a digital micromirror device (DMD). The size of the digital light processing (DLP) area is $14.0 mm \times 10.5 mm$ , and the number of pixels is $1024 \times 768$ . The area displayed by the object is approximately $10 mm \times 10 mm$ . By ensuring the illumination and observation angles, the direct reflection from the area out of the object on DMD’s front surface is minimized. With a 50 mm focal length lens, a polarization camera (DYK 33UX250, the ImagingSource) is focused onto the VD. The polarization camera utilized in this work employs a four-element microlens array to simultaneously capture speckle patterns at four distinct polarization angles (0°, 45°, 90°, and 135°). The relay wall is made of latex paint.

4. Results and Discussion

Using the above system, we image four hidden objects: “N”, “L”, “O”, and “S” of approximately $20 mm \times 20 mm$ in size. Through an efficient single-shot polarized speckle illumination strategy, hidden objects are encoded by four uncorrelated polarized speckles. The power spectrum of the object is obtained through intensity acquisition by the polarization camera. After correcting the collected polarized speckles using Eq. (9), the autocorrelation estimate of the hidden object is calculated based on the autocorrelation estimation method composed of Eqs. (7) and (8). By employing phase retrieval algorithms such as the hybrid input-output (HIO) algorithm^[24], deep learning methods^[25], and the alternating-direction-method-of-multipliers-based method^[26], the hidden object can be reconstructed. Reconstructions in this Letter use the HIO algorithm.

Replacing the mechanical scanning with polarization modulation avoids the multiple exposure process of the mechanical scanning method and ensures the stability of the system geometry. The acquisition time is reduced and the acquisition efficiency is improved, ensuring reconstruction consistency and robustness against perturbations. The average object structural similarity index measure (SSIM) reconstructed by the proposed PDC-NLOS is about 0.76, which is better than the reconstruction results with SSIM of about 0.6 of single-shot NLOS imaging based on chromato-axial differential correlography^[21]. PDC-NLOS illuminates indirectly the hidden object through encoding speckles. However, the albedo of the hidden object is uneven, and the speckle intensity fluctuates, which leads to ambiguity in absolute intensity recovery, manifesting as color bar inconsistencies in Fig. 4. This noise propagates into phase retrieval, causing local intensity estimation errors that reduce SSIM.

Figure 4.Experimental results of the PDC-NLOS. (a1)–(a4) Four-polarization-channel intensity images collected by a polarization camera; (b1)–(b4) hidden objects to be detected; (c1)–(c4) autocorrelation estimates; (d1)–(d4) reconstruction results using the HIO algorithm.

Download full size

View all figures

With a USAF target shown in Fig. 5(a), we image vertical and horizontal stripes to measure the imaging resolution of the system. As shown in Figs. 5(c1) and 5(c2), it is not difficult to find that the proposed PDC-NLOS imaging method can resolve the stripes at millimeter resolution. The imaging results indicate that the PDC-NLOS imaging system has a millimeter-level resolution, but the vertical resolution is better than the horizontal resolution. The curves in Fig. 5(d) represent the power spectrum intensity distributions collected by the system in the vertical and horizontal directions for horizontal and vertical stripes as objects. The contrast of the power spectrum information for horizontal stripes in the vertical direction is greater than that of vertical stripes in the horizontal direction, which confirms that the vertical resolution of the system is better than the horizontal resolution. From the perspective of system construction, the laser irradiating from the VS is obliquely incident on the object surface. After emerging from the object’s surface, it is collected obliquely by the VD, which compresses the horizontal pixel distribution of the object. Object pixels in the vertical direction do not suffer from compression, so the horizontal resolution will be lower than the vertical resolution.

Figure 5.Resolution test experimental results of the PDC-NLOS. (a) A USAF target; (b1), (b2) autocorrelation estimates of the vertical and horizontal stripes; (c1), (c2) reconstructions using HIO reconstruction; (d) intensity distribution of the power spectrum in the direction perpendicular to the object.

Download full size

View all figures

It should be noted that the imaging results of the proposed method still have room for improvement. Due to the incomplete sampling of the Fourier amplitude by the camera, it is difficult to reconstruct more complex objects using the traditional HIO algorithm. Recent advances in unsupervised deep learning frameworks offer promising solutions to improve phase retrieval fidelity under noisy conditions, which could be integrated with our polarization-based encoding strategy in future work^[17].

In the actual scene, the hidden object may be diffuse or have uneven polarization. Diffusion reduces the coherence of speckle, resulting in the inability to encode objects. Objects with uneven polarization may lose some information. When considering the albedo and polarization retention ability of an object, the albedo function of the object in Eq. (2) will be replaced by the following equation instead of the binary function that is originally related to the shape of the object: $r_{O} (x, y) = \sum_{θ_{n} = 0}^{2 π} r_{O} (x, y, θ_{n}) .$ (10)

Under different $θ_{n}$ , $r_{o}$ will have a grayscale distribution that is related to the polarization state distribution of the object. When the albedo of an object material changes significantly under different polarization states, polarization redundancy can support PDC-NLOS for object reconstruction. Otherwise, structures with significant changes in albedo under different polarization states of the object will be lost.

5. Conclusion

In conclusion, this study proposes a PDC-NLOS imaging method for high-resolution imaging of hidden objects around corners. We have demonstrated the imaging capability of the proposed method with the actual experimental setup. With the designed single-shot polarized speckle illumination strategy, PDC-NLOS can efficiently encode the hidden objects without mechanical scanning. With a single laser source and a polarization industrial camera, stable encoding and reconstruction of hidden objects have been achieved in low-cost hardware systems. PDC-NLOS provides valuable insights for the stable and real-time development of steady-state NLOS imaging.

Category: Imaging Systems and Image Processing

Received: Feb. 24, 2025

Accepted: Apr. 18, 2025

Published Online: Aug. 1, 2025

The Author Email: Enlai Guo (njustgel@njust.edu.cn), Jing Han (eohj@njust.edu.cn)

DOI:10.3788/COL202523.081104

CSTR:32184.14.COL202523.081104