Non-line-of-sight (NLOS) imaging is a computational imaging technique that uses a reflective relay surface to capture images of objects outside the detector’s field of view (FOV). An observer can use a reflective surface to view scenes behind obstacles or around corners, a technique that has extensive application value in scenarios such as firefighting rescue, reconnaissance, and cave exploration. Reconstructing and detecting NLOS objects under strong scattering has significant scientific, social, and economic value. However, the randomness of the scattering characteristics of relay surfaces and the uncertainties regarding the object make NLOS imaging very challenging.

In terms of detection methods, solutions for NLOS object reconstruction include time-of-flight (TOF) detection schemes and correlation-based detection, among others. The specific devices utilized include single-photon avalanche detectors (SPADs), conventional cameras, infrared cameras, and hyperspectral cameras [1–9]. In terms of computational methods, reconstruction algorithms for NLOS objects include speckle correlation calculations [10–13], deconvolution operations [14–17], Fourier transforms [18,19], and deep neural network modeling, among others [20–25]. The NLOS optical information collected by detectors through relay surfaces is subject to significant interference and randomness. Although there are solutions that involve high costs to enhance detector sensitivity and data richness, challenges such as long imaging times, complex algorithms, and poor imaging quality still persist.

From the perspective of scattering characteristics, the measured light field couples scattering information from both the object and the relay surface in NLOS imaging. The core objective of NLOS imaging lies in separating the target light component from this coupled information and achieving its reconstruction. Material surface optical scattering properties are typically described using the bidirectional reflectance distribution function (BRDF). Xu et al. [26] incorporated the BRDF and proposed a deep neural framework based on feature extraction and enhancement. This framework enables noise suppression and detail preservation in NLOS object information. Liu et al. [27] used the advantages of a confocal system and a spot scanning detection method and reported an NLOS imaging method based on a heteromorphic relay surface, which has rich scattering characteristics. Sasaki et al. [28–30] derived a precise relationship between the light field of an object and the scattered light field, incorporating the BRDF of the relay surface. Ultimately, they achieved target reconstruction by solving the Fredholm integral equation. However, the BRDF has limitations when describing relay surfaces. Lin et al. [31] utilized a conventional camera and demonstrated through inverse calculations of the BRDF that Lambertian scattering relay surfaces pose significant challenges for passive cameras.

From the analysis of complex signal reconstruction algorithms, recent advancements have demonstrated that leveraging artificial intelligence’s computational capabilities to perform inverse computation on scattered light information has emerged as a paradigm in computational optics [32–35]. In addressing the reconstruction of severely degraded image information, neural networks primarily leverage the generative and denoising capabilities of models. Liang et al. [32] proposed a framework based on a generative adversarial network (GAN) named NLOS-GGAL. In this approach, the discriminator processes in-line-of-sight images and the generator produces non-line-of-sight images, thereby enabling the generation of non-line-of-sight objects. However, generative models are limited to producing targets they have been trained on, which restricts their applicability in imaging tasks. On the other hand, neural network models learn data features to eliminate interference and achieve image reconstruction. He et al. [1] designed a variational autoencoder and achieved object reconstruction in a scene with a paper man and white wall. However, whether the generated results are consistent with the real scene remains to be discussed. Metzler et al. [36] employed spectral estimation theory to design a denoising model, utilizing deep neural networks for noise removal to achieve high-resolution non-line-of-sight imaging. Chen et al. [37] designed a deep neural network on the basis of physical characteristics for reconstruction, but the scattering characteristics of the relay surface or the object were not discussed in detail. Deep learning methods exhibit strong adaptability in describing complex physical phenomena. However, they still face challenges such as generalization issues, data acquisition problems, and training methodology limitations.

From the detector configuration perspective, NLOS imaging systems typically utilize SPDA devices (single-photon detection arrays), conventional cameras, and infrared cameras as key sensing components [38–41]. Zheng et al. [42] proposed a method based on segmented ellipsoidal interpolation (SEI), which transforms non-confocal measurements into semi-confocal measurements to improve temporal resolution and imaging quality using SPAD. Ding et al. [43] introduced a reconstruction model based on curvature regularization to address the inverse problem of sparse sampling data. Xiao et al. [44] developed a hybrid super-resolution network that significantly reduces the number of scanning points without compromising imaging quality. Geng et al. [45] designed a manifold-embedded encoder to address the reconstruction quality of passive NLOS imaging for conventional cameras. Liu et al. [46] proposed a dual-channel input deep neural network based on an infrared polarization camera, incorporating an attention mechanism to enhance NLOS imaging quality. Czajkowski and Murray-Bruce [47] used some matte relay surfaces and ordinary 2D cameras to implement 3D reconstruction of hidden scenes. Saunders et al. [6] introduced a novel approach by reconstructing non-line-of-sight objects using opaque objects with visible contours as occlusions. The high-frequency information at the edges of the occluders forms a detection aperture, enabling target detail recovery through inverse calculations. However, passive NLOS detection methods, which only receive scattered light spots, face significant challenges in reconstructing spatial object information.

This study employs a 3D depth imaging camera based on a photonic mixer device (PMD) [48], which captures the phase difference between modulated illumination light and signal light to perform depth ranging imaging through a TOF camera. This camera is capable of acquiring both 3D depth data and intensity data of the target in a single frame. Heide et al. [49] utilized a TOF camera for NLOS imaging by introducing a regularization term to strengthen the connection between the light signal transmission matrix and the speckle correlation matrix, transforming the inverse problem into a nonlinear optimization problem. Kadambi et al. [50] proposed a framework for an NLOS imaging model based on a TOF camera, establishing a relationship between the imaging angle and the BRDF of the reflective surface, and achieved NLOS imaging using a compressed sensing algorithm. Although the resolution of the imaging results is relatively low, it demonstrates the potential of consumer-grade TOF cameras for NLOS imaging. Consequently, in this study, the scattering combination relationship between the object and the relay surface is investigated in detail, and a scalable scattering mapping (SSM) theory is presented to delineate the scattering mapping process from a clear image to a degraded image. By utilizing a TOF camera, a substantial volume of image data under various scene settings is extensively collected. Ultimately, a high-quality 3D depth imaging of NLOS objects is achieved by employing artificial intelligence models. This study aims to address the prevalent issues in previous NLOS imaging systems and algorithms, such as high economic costs, algorithmic complexity, and subpar imaging quality.

NLOS imaging systems can employ various illumination modalities according to their imaging principles. When the illumination source and the detector are on the same side, TOF systems must address the issue of triple-bounce scattering of the light signal. Conversely, when the illumination source and the detector are on opposite sides, the light information received by the detector undergoes two scattering events, thus avoiding interference from multiple scattering events, as commonly observed in passive imaging methods. The TOF camera employed in this study, which is based on a PMD detector, is an active illumination 3D imaging camera. The illumination source is non-coherent light of a single wavelength, and the illumination system applies modulated light to fully illuminate the target. Depth is derived by measuring the phase difference between the illumination signal and the received signal, thereby obtaining the depth data of the object. Since the light source is non-coherent modulated light, the illumination from a point source on the same side is equivalent to scattering illumination on the opposite side. Our proposed configuration employs illumination on the opposite side. The structural comparison between NLOS imaging and line-of-sight imaging shows that the reflective relay surface is equivalent to a scattering medium between the object and the detector, as shown in Fig. 1.

The illuminance c(τ) recorded at each pixel of the PMD detector reflects the correlation operation between the illumination modulated light signal s(t) and the received light signal g(t). The distance to the target is calculated by delaying the phase of the modulated light signal. Since all pixels operate simultaneously in the same manner, this method generates a depth image of the target. However, the PMD detector is sensitive to the optical scattering characteristics of the object, and different scattering characteristics can cause nonlinear variations in imaging accuracy. In addition, the PMD detector cannot capture purely specular objects. The imaging model can be expressed as follows [48]: $$\begin{gathered} c(τ)=s(t)⊗g(t)=limT→∞1T∫−T2+T2s(t)g(t+τ)dt, \\ φtof=arctan(c(τ3)−c(τ1)c(τ0)−c(τ2)),d=c0φtof4πf, \end{gathered}$$(1)where c(τx) represents the value of the correlation function, where τx denotes the phase difference of the modulated illumination light signal. The target distance is determined by utilizing four fixed illumination phase differences; f denotes the frequency of the modulated light, T signifies the period of one cycle of the modulated illumination signal, c0 is the speed of light, and d is the target distance value calculated for a particular pixel. In NLOS imaging, assuming the scattering effect of the relay surface is F, the returned light signal g′ received by each pixel is the combined result of the clear returned signal g and the scattering characteristic F, which can be expressed as g′=g⊗F. Therefore, analyzing the combined scattering characteristics of the object and relay surface is an important issue.

Based on the fundamental principles of optical path reflection, the proportion of the specular component in the optical characteristics of the object and the relay surface is examined. When the relay surface is specular and the object is Lambertian, the detector records a specular image of the object. However, as the scattering characteristics of the relay surface transition from purely specular to purely Lambertian, the object’s light information reaching the detector gradually becomes scattered and blurred. In the resulting degraded images, noise interference increasingly dominates. On the other hand, when the scattering characteristics of the Lambertian object surface gradually shift from purely Lambertian to purely specular, the speckle pattern on the Lambertian relay surface reflects the object’s light information. Therefore, the raw data from the detector is a coupled outcome of the scattering characteristics of both the object and the relay surface. Similar to image contrast, if the scattering characteristics of the object and relay surface differ significantly, decoupling the raw imaging is relatively straightforward; conversely, when their scattering characteristics are identical, decoupling becomes extremely challenging, such as in cases where both exhibit specular reflection or both exhibit Lambertian scattering. Figure 2 presents the imaging results of objects and relay surfaces with different scattering characteristics using intensity images from a TOF camera, demonstrating this scattering coupling phenomenon.

The curves in Fig. 2 represent simplified diagrams of the BRDF for objects A, B, and C, as well as for relay surfaces a, b, and c. The sharper the curve is, the greater the proportion of specular reflection in the material’s surface characteristics. Subjectively, in the case of the Aa combination, the raw data appears indistinguishable to the naked eye, indicating a tight coupling of the scattering characteristics of the two surfaces. Conversely, the Cc combination results in partial pixel saturation in the detector due to the intensity of the returning light. The other combinations in Fig. 2 demonstrate that the NLOS object light signals follow a statistical pattern overall. The more specular the relay surface is, the more the imaging tends towards specular reflection, and the more specular the object is, the more the imaging tends towards reflective imaging.

The BRDF describes the intensity distribution of light exiting a material surface when illuminated by a point source. Let the scattering characteristic functions of the object and the relay surface be fobj and frs, respectively. The irradiance Ecam on the camera sensor surface, after the light information from a point on the object passes through the relay surface, can be expressed as follows: Lobj→rs(θin,θout,φout)=fobjEcosω,(2)Ers=Lobj→rscosω′r2,(3)Lrs→cam(θin′,θout′,φout′)=frsErscosω′,(4)Lrs→cam(θin′,θout′,φout′)=1r2frsfobjEcosωcos2ω′,(5)Ecam=1k21r2frsfobjEcosωcos2ω′cosω′′,(6)where Lx and Ex mean radiance and irradiance. The subscripts obj→rs, rs→cam denote the propagation paths of the light information; light travels from the object to the relay surface, and from the relay surface to the camera. E denotes the irradiance from the light source to the object point. θx and φx denote the azimuth and zenith angles of the incident and transmitted light signals, respectively. k and r represent the distances from the object and the camera to the relay surface, respectively. As evident from the formula, the light information undergoes two scattering events before reaching the TOF camera sensor: one due to the object’s scattering characteristics and the other due to the relay surface’s scattering characteristics. The object’s scattering characteristic fobj determines the distribution of light information on the relay surface, while the relay surface’s scattering characteristic determines the spatial distribution of the object information on the sensor side. The irradiance Ecam in Eq. (6) received by the detector corresponds to Cx in Eq. (1), where x is generated by delaying the modulated illumination signal, thereby enabling the acquisition of 3D data. Figure 3 visualizes the propagation characteristics of the light information. The presence of more Lambertian scattering components on the surfaces of both the relay surface and the object leads to a broader distribution of effective light information from the object. Therefore, we propose an SSM model to address the inverse problem.

Based on the combined analysis of the scattering characteristics of the object and relay surface, the object’s light information undergoes varying degrees of diffusion as it passes through the relay surface. To address this, we have designed an SSM model for inverse computation. As shown in Fig. 4, the object information is captured by the detector under different scattering conditions.

After the optical signal reflected by point a of the object is incident on the relay surface, the light is reflected into the corresponding pixel a′ of the detector. During specular reflection, point a has a one-to-one correspondence with point a′. When the relay surface is non-specular, the pixels surrounding pixel a′ can also receive light energy emitted from point a, which spreads and attenuates outward from the center. This phenomenon can be referred to as a “pixel edge effect.” Similarly, the light emitted from point b adjacent to point a is scattered by the relay surface and falls on pixel b′. When the scattering characteristics of the relay surface differ, the energy and range received by the surrounding pixels of a′ and b′ will vary, and the irradiance from these pixels will overlap. When all pixels operate simultaneously, the result is image degradation. Therefore, the irradiance received by a specific pixel p will be the superposition of the “edge effects” from all surrounding pixels. Equation (6) can be rewritten from the perspective of a pixel as follows: $$\begin{gathered} Eres(x,y)=Cfrs(lobj→rs,vrs→cam)fobj(ls→obj,vobj→rs) \\ +C∑i∈Rfrs(lobj→rs′,vrs→cam)fobj(ls→obj,vobj→rs′)+ε, \end{gathered}$$(7)where C=1k2r2Ecosωcos2ω′cosω′′.(8)

lx and vx represent the incident and outgoing directions of the light information, respectively. The SSM model describes the distribution range of object light information on the detector pixels after being scattered by the relay surface. The effective light information is distributed across different pixel ranges based on the scattering characteristics of the relay surface and object. The wider the distribution of effective information is, the weaker the irradiance received by the pixels and the more susceptible it is to interference. During the reconstruction process, for cases where the effective information is distributed over a larger range, more pixels are utilized to capture potential object information.

On the basis of SSM theory, the speckle information collected by a detector is the result of the joint action of the scattering characteristics of the relay surface and object, which are coupled, while the reconstruction of a clear image is an inverse process of coupling. Due to the presence of uncertain incident and outgoing light angles in Eq. (7), direct computation is relatively challenging. Therefore, based on its descriptive characteristics, we assume that the degraded image is the result of the convolution of a clear image with a certain scattering effect. We have designed a deep neural network with variable input and output dimensions to decouple scattered images into clear images. Clear and ordered object light information diffuses after scattering from the relay surface, and the size of the scattering-degraded image data is larger than that of the clear image data as shown in Fig. 5.

The input and output of the SSM-based NLOS image reconstruction network model are different sizes. The proposed model establishes a mapping relationship from a large scattering-degraded 3D image to a small clear 3D image. For a determined relay surface, a mapping from the speckle image to the clear image is established to learn the mapping characteristics of the relay surface and object scattering imaging, thus realizing the decoupling of the scattering characteristics of the relay surface and the object. The network model uses a multichannel, multibatch convolution operation to extract the features of NLOS objects in the scattered images. The reconstruction process is divided into downsampling feature encoding and upsampling image decoding and reconstruction. The downsampling operation process realizes the fusion of scattering features, and the data features before fusion are screened by the channel attention mechanism and then passed to the upsampling process. The upsampling operation process decodes the scattering features and finally reconstructs the output result. When a new scattering-degraded image is input into the model, a clear image is obtained after reconstruction. In the process of up- and downsampling, the convolution operation is the result of summing the product of each pixel of the image and the convolution kernel, which is consistent with the analysis theory in Eq. (7), where convolution can be considered as the scattering property of the relay surface, and we use neural networks to characterize it.

The SSM reconstruction method is aimed at relay surfaces with fixed scattering characteristics. First, before application, scattering-degraded images from the relay surface and corresponding clear images are used as the training set. Therefore, in practical applications, the material of the reflective relay surface in the scene needs to be modeled in advance to form a database for application in real scenes.

The experimental setups of scene and objects are shown in Fig. 6. The imaging device was a TOF camera (model OPT8241CDK from TI). The direction of the illumination light was adjusted to ensure that the object surface was fully illuminated. The pixels of the object part in the FOV were used in data processing. In the laboratory setup, two types of objects were selected. The plaster statue is a uniform Lambertian body with complex geometric characteristics, and the plastic letters are simple objects with some specular components. In the real-world setup, a relatively smooth painted door and a white wall close to Lambertian scattering were chosen as relay surfaces and the object was common clothing to achieve NLOS 3D imaging similar to that of people. To establish the scattering reconstruction model for relay surfaces and objects through SSM theory, first, a large amount of training data must be collected; then, the model must be validated on the new data. The laboratory setup was used to test the feasibility of SSM theory, and the real-world setup was used to apply the theoretical method.

In the laboratory setup, two common scattering media and a standard diffuse reflective plate were chosen to test the adaptability of SSM theory to different relay surfaces, where the selection of the relay surface was arbitrary. The two scattering media used were a polished acrylic plate (polymethyl methacrylate, PMMA) and a plastic plate (polypropylene, PP), whereas the diffuse reflective plate was a standard Lambertian scattering plate with a reflectance of 98% at 850 nm. The distributions of the scattering characteristics of two scattering media are shown in Fig. 7. In the real-world setup, the scattering characteristics of the door had some specular components, whereas the white wall surface coated with white paint had Lambertian scattering characteristics.

Using the aforementioned relay surface and object, data acquisition was conducted in both laboratory environments and real-world scenarios. Speckle images of the object on the relay surface were captured with a time-of-flight (TOF) camera as the reconstruction object data. After replacing the relay surface with a flat mirror, clear images were acquired under identical positional configurations to serve as ground truth data. The object was mounted on a rotary platform for imaging. As the object completed a full rotation (approximately 800 frames captured by the TOF camera), data augmentation was implemented through image rotation to obtain multi-pose object representations. Model training was performed using an NVIDIA GeForce RTX 4090 GPU, with code developed in MATLAB R2023b. Tables 1 and 2 summarize the data acquisition specifications and model training parameters.Table 1.

Training Data Set

The imaging reconstruction method proposed in this work is divided into training and application stages in practical applications. First, a model for the potential reflective relay surfaces in the application scenario is established. Then, blurred images are collected through potential relay surfaces, whereas clear images are collected through mirrors. These data are used to learn the scattering characteristics of the potential relay surfaces. Then, in the corresponding scene, the trained model can be directly used for NLOS imaging.

The polished plastic plate and acrylic plate had a certain specular reflection component, and when they were used as the relay surfaces mentioned in Fig. 7, direct visual observation posed challenges for the human eye. The first rows of Fig. 8 show the unprocessed data collected by the TOF camera, showing the severely degraded results. On the basis of the design in Fig. 5, clear and degraded images of the objects were collected through a plane mirror and the relay surfaces, respectively, and were used as the training set for mapping from degraded to clear. Then, the trained model was used to reconstruct data from the test set. The 3D images in the second row of Fig. 8 were reconstructed from the degraded images in the first row. Because the plastic plate and acrylic plate had more specular components and obvious features, the SSM model used input and output images of the same size during training and could decouple the scattering characteristics of the object from those of the relay surface. The number of data in the test sets for the two types of relay surfaces is approximately 4000. To verify the model’s generalization ability, all 4000 test images were fully reconstructed. Quantitative evaluation was conducted using MSE and PSNR. Figures 8(d) and 8(e) display the comparative metrics between before-reconstruction and after-reconstruction states for all test images. The data have been systematically sorted to clearly identify the optimal MSE and PSNR values achieved through the reconstruction process. Therefore, the model had a strong decoupling ability for data that were not used in the training.

To further test the feasibility of SSM theory, a pure Lambertian scatterer was used as the relay surface, and the Lambertian plaster statue and plastic letters with specular components were the objects. Figure 9 shows that when the plastic letters were used as the objects, the unprocessed data had inconspicuous object characteristics, which occurred because the specular component of the letters reflected the illumination light on the relay surface. However, due to the scattering characteristics of the object, the object after scattering was larger than the actual object; a scattering-degraded image of 192×192 size was used as the input and a clear image of 96×96 was used as the output to train the SSM model. The images in the second column of Fig. 9 show that the reconstruction results were very good, which was due to the large difference between the scattering characteristics of the object and those of the relay surface. The combination of Lambertian objects with non-Lambertian reflectors is equivalent to the combination of Lambertian reflectors with non-Lambertian objects. In the end, an experiment was conducted under the worst conditions; that is, both the object and relay surface were Lambertian.

When the object and relay surface were both Lambertian, as shown in Fig. 10, the unprocessed 3D image was almost completely pure noise; this noise was from the surroundings of the relay surface. However, according to the analysis in Fig. 3, light information from an object can still be obtained from an occluder. Thus, the mapping relationship from the clear image to the scattered image was established, and the reconstruction results are shown in the second column in Fig. 10. To increase the richness of the data, in the Lambert-Lambert combination experiment, in addition to a single object, a combination of multiple objects was also used. The worst imaging condition is when the object and the relay surface are both Lambertian. The MSE and PSNR of 4000 reconstructed results show that the model’s average performance improvement is not as much as the reconstruction processing results of the aforementioned object-relay surface combination. The SSIM was introduced to conduct a supplementary assessment of the model’s reconstruction performance, and the evaluation results demonstrated that the SSM model exhibits a strong decoupling ability in the Lambert-Lambert combination setups.

The experimental results for two different objects reveal that when the relay surface experienced Lambertian scattering, the scattering characteristics of the object had a considerable effect on the imaging results. An increase in the specular component of the object enhanced the imaging result, and vice versa. The results on the test set show that the SSM model still had some decoupling ability when processing NLOS images under harsh conditions and could reconstruct data from the test set after learning the scattering characteristics of the relay surface. The proposed method was then further applied to a real-world setup to test its adaptability to natural objects.

The SSM model yielded excellent reconstruction results in the laboratory, so we further tested it on a real-world setup. A brightly painted door in the hall and a white wall were chosen as the reflective relay surfaces, and the object was ordinary clothing to simulate NLOS 3D imaging of people. The first row in Fig. 11 shows the unprocessed images collected by the TOF camera. The second and third rows are the reconstruction results and the true object images for comparison, respectively. The surface of the door has a paint coating that includes a specular component; thus, the reconstruction effect is superior to that of the wall surface. The diffuse characteristics of the lime plaster wall are approximately equivalent to those of a Lambertian surface; however, the wall surface still has a small specular component. Hence, the reconstruction result is better than that in Fig. 10. The experimental results show that the SSM reconstruction model performed very well on the real-world scenes. The SSM model was able to achieve NLOS image processing. The SSM can indeed learn the scattering characteristics of the relay surface.

We present an innovative scalable scattering mapping model (SSM) based on deep neural networks to analyze the NLOS imaging process from light propagation dynamics. Our framework achieves decoupled reconstruction of object-relay surface coupled scattering information by characterizing optical propagation properties across different material interfaces. A critical observation reveals that larger differential scattering signatures between objects and relay surface enhance NLOS imaging fidelity, which we validate through systematic experiments.

Employing a low-cost TOF 3D camera, our method successfully reconstructs NLOS objects from scattered light patterns on reflective surfaces. Despite severe image degradation caused by complex surface scattering effects, speckle patterns retain crucial object information. Through benchmark experiments using a gypsum sculpture with uniform scattering properties, we demonstrate successful Lambertian interface reconstruction. Further practical validation employs two distinct relay surfaces: a specular-dominated painted door and a near-Lambertian white wall, achieving comparable reconstruction quality using ordinary clothing objects.

However, current limitations emerge when handling untrained object-relay surface combinations with dissimilar scattering characteristics. To address these constraints, we propose two data expansion paradigms: (1) scene-specific training encompassing all potential object-mediator pairs within objected environments; (2) object-centric training covering diverse application scenarios for specific objects. Inspired by the iterative training paradigm in large language models, we envision continuous dataset expansion through active learning strategies to progressively achieve intelligent NLOS imaging systems capable of handling complex real-world scenarios.

In this study, on the basis of the combination of the scattering characteristics of the relay surface and the object in NLOS imaging, a theoretical scalable scattering mapping model (SSM) is proposed, and a deep neural network for feature extraction is designed. After sufficient training on objects and relay surfaces, NLOS 3D imaging using a TOF camera can be achieved. The experimental results show that the proposed method yields excellent imaging results, can be applied to natural scenes, and is very robust.