Augmented reality (AR) is revolutionizing areas such as education, entertainment, and medical services by merging real-world and computer-generated content^[1–7]. As the primary sensor for environment perception, cameras are a critical component in AR devices^[8–11]. However, their inherent opacity conflicts with the see-through nature of AR displays. To address this, several transparent camera technologies have been developed. These can be broadly categorized into two paths: one using inherently semi-transparent photodetectors, for example, by stacking graphene light-sensing layers on high-transmission substrates^[12], and the other employing conventional opaque photodetectors whose peripheral circuits are concealed using beam-splitting elements^[13] or light modulators^[14]. Another promising direction is computational imaging, which leverages hardware like luminescent concentrator films^[15], windows with roughened edges^[16], and holographic optical elements^[17–22]. Despite these innovations, a common bottleneck remains: the imaging resolution is often constrained by low-density photodetector arrays or the necessity for computationally intensive reconstruction algorithms.

A recent hardware platform, LightguideCam, offers a compact and flexible solution for a high-resolution see-through computational camera by guiding a portion of the incoming light to an image sensor, forming a blurred image^[23]. However, reconstructing a clear image from this blurred measurement using traditional model-based iterative algorithms like the fast iterative shrinkage-thresholding algorithm (FISTA)^[24–26] is computationally demanding and too slow for real-time applications. This challenge has motivated a shift toward data-driven methods, as deep learning has become a powerful tool for solving such inverse problems in computational imaging^[27–30]. Once a neural network is trained, it can perform image reconstruction at very high speeds^[31,32]. The field has rapidly evolved from early architectures like U-Net^[33,34] to more efficient physics-informed neural networks^[35,36], which combine the advantages of data-driven and model-based approaches to improve performance and data efficiency^[37–41].

In this work, we address the reconstruction speed and quality challenge for the LightguideCam. We employ a physics-informed deep learning framework that replaces the slow model-based FISTA algorithm. This new approach achieves a dramatic increase in reconstruction speed and a significant improvement in image quality. The simple design of the LightguideCam, combined with the high-performance reconstruction network, offers a compelling solution for seamless integration into AR display devices, with great potential for applications such as eye-gaze tracking and eye-position-perspective photography.

The core of the see-through camera system is a lightguide that enables simultaneous viewing of a scene and imaging of it, as illustrated in Fig. 1. The system can be described by a forward model that accounts for the spatially varying point spread function (PSF) introduced by the camera optics [Fig. 1(a)]. For this model, the object image is treated as a 2D array of pixels, v[x,y]. Each pixel has a corresponding PSF, h[x′,y′;x,y], that describes its projection onto the camera sensor, where [x′,y′] are the image space coordinates. Assuming incoherence between object pixels, a measurement b[x′,y′] can be expressed as a linear combination of the PSFs from all object pixels: b[x′,y′]=∑x,yv[x,y]h[x′,y′;x,y]=Av.(1)Here, the matrix A represents the system’s complete response, mapping the object v to the measurement b. The inverse problem, which is the central challenge, is to recover v given b and an estimate of A [Fig. 1(b)]. In the previously published work on LightguideCam^[23], the iterative FISTA algorithm was applied to solve this problem^[24]. However, reconstructing a single image required several minutes, which is far too slow for real-time applications. To overcome this limitation, we employ a deep-learning-based approach for high-speed, high-quality reconstruction.

Training a deep neural network (DNN) requires a large dataset of paired measurements and ground truth images. The experimental setup used for this data acquisition is shown in Fig. 2(a). A high-resolution OLED display is placed at a working distance of 70 cm from the LightguideCam. A partially reflective mirror array lightguide is employed in the system, equipped with a concave mirror for focusing. A cooling fan is used to maintain a stable sensor temperature, preventing an increase in thermal noise during the lengthy data capture process. As outlined in Fig. 2(b), images are sequentially loaded onto the display to serve as the ground truth, and the corresponding distorted images are captured by the camera’s image sensor. For the training dataset, we used approximately 6000 images sourced from the Div2k^[42] and Flickr2k^[43] datasets, with some images augmented by rotation. The captured data were randomly split, with 90% used for training and 10% for validation and testing. While it is possible to simulate training data using the forward model^[23], using experimentally captured data ensures that the network learns to invert the true physical properties of the system, leading to superior reconstruction quality.

For high-speed reconstruction, we use a physics-informed neural network, the Multi-Wiener Net (M. W. Net), based on a multi-Wiener deconvolution approach^[36], whose architecture is detailed in Fig. 3(a). The network consists of two main stages. First, a deconvolution layer processes the input measurement b to generate multiple intermediate solutions. This layer contains a bank of M parallel Wiener filters. For each filter i, the intermediate reconstruction in the Fourier domain, V^i(u,v), is calculated as V^i(u,v)=Hi*(u,v)B(u,v)|Hi(u,v)|2+λi,i=1,2,…,M,(2)where (u,v) are the frequency space coordinates, B is the Fourier transform of the measurement, Hi is the Fourier transform of the ith PSF, and λi is a corresponding regularization parameter. Crucially, while the PSFs are initialized by capturing the system’s response to point sources across the field of view (FoV) [Fig. 3(b), left], both the PSFs (Hi) and the regularization terms (λi) are learnable parameters. They are updated during training, allowing the network to refine the physical model and better approximate the complex, shift-variant nature of the optical system, as seen in the transition from initial to trained PSFs in Fig. 3(b).

In the second stage, the M intermediate reconstructions are transformed back to the spatial domain via an inverse Fourier transform and are then concatenated and passed to a convolutional neural network (CNN). This CNN, based on the U-Net architecture^[33], is modified to accept multiple input images and regress to a single output image. Its function is to intelligently fuse the information from all intermediate images, combining regions of high fidelity from each and suppressing artifacts to produce the final clean reconstruction.

The specialized architecture of the M. W. Net is the direct reason for its superior performance. Unlike purely data-driven “black box” models, it incorporates a physics-informed deconvolution layer where each Wiener filter learns to invert the blur for a specific region of the image. The subsequent U-Net is then tasked with the simpler more well-defined problem of fusing these intermediate, high-fidelity reconstructions and suppressing noise. This structured approach, which explicitly models the system’s spatially variant physics, is the key to achieving superior reconstruction fidelity, particularly at the edges of the FoV where traditional methods often fail. To assess the performance gain by including the physics priors in the M. W. Net, a standard U-Net was trained on the same dataset and with the same hyperparameters for comparison. For the test dataset evaluated in Fig. 4, the mean peak signal-to-noise ratio (PSNR) was reduced by 2 dB.

The entire network is trained end-to-end using the Adam optimizer to minimize the mean squared error (MSE) between the final reconstruction and the ground truth image. We trained a separate network for each color channel. For an image size of 2048 pixel × 2048 pixel, each network has 51 million learnable parameters. Training was performed on an NVIDIA RTX A6000 GPU, taking 31 to 56 h for 30 to 60 epochs. The loss curves in Fig. 3(c) demonstrate successful convergence for all three color channels. The training process was halted as soon as the validation loss did not improve for 10 consecutive epochs to prevent overfitting. In conjunction with the diverse training dataset, this enables good generalization ability by the network.

To evaluate the performance of our proposed method, the trained M. W. Net was benchmarked against the raw sensor measurement and the previously used iterative FISTA algorithm^[23]. The comparison was conducted on a test dataset not seen during training, with the results summarized in Fig. 4 and Table 1.

The quantitative metrics, illustrated in the violin plots in Figs. 4(a)–4(c), show a dramatic improvement in image quality. As detailed in Table 1, the M. W. Net achieves a mean PSNR of 20.32 dB, a significant increase of approximately 7.7 dB over FISTA (12.63 dB) [Fig. 4(a)]. Similarly, Fig. 4(b) shows the mean structural similarity index measure (SSIM) score jumps to 0.68 for our method, compared to 0.42 for FISTA. This high degree of structural fidelity is further confirmed by the Pearson’s correlation coefficient (CC) shown in Fig. 4(c), which increases to a mean of 0.92. These quantitative gains are reflected visually in the example reconstruction in Fig. 4(d). Our network’s output is significantly sharper and more detailed than the FISTA reconstruction, appearing much closer to the ground truth. The superior accuracy is explicitly visualized by the residue images in Fig. 4(e), where the substantially darker result for the M. W. Net indicates a much smaller reconstruction error. Furthermore, the pixel-wise SSIM map in Fig. 4(f) reveals that the M. W. Net provides uniformly high reconstruction quality across the entire FoV, whereas the FISTA method exhibits noticeable degradation in quality, particularly toward the edges.

Beyond image quality, the most transformative advantage of the M. W. Net is the reconstruction speed. As shown in Table 1, the iterative FISTA algorithm required an average of 1067 s (nearly 18 min) to reconstruct a single image. In stark contrast, our neural network completes the task in just 0.273 s. This constitutes an acceleration by a factor of almost 4000, enabling real-time applications with imaging rates of up to 4 frames per second.

To test the network’s performance on three-dimensional scenes, we captured images of objects placed at varying depths, as shown in Fig. 5. The scenes include playing cards at discrete depths and a continuous object with depth variation. Since the network was trained using a dataset captured at a single plane (70 cm), the reconstructions sharply resolve objects located at or near this training depth, showing the generalization ability of the network. Objects at other depths remain blurry, demonstrating the depth-selective nature of the current network. This highlights a potential avenue for future work, where extending the network by capturing PSFs from multiple depth planes to initialize additional learnable Wiener filters and using a 3D-U-Net to combine the intermediate outputs could enable digital refocusing across different depths^[36].

This work presents a physics-informed deep learning framework that enables high-speed and high-fidelity image reconstruction for the LightguideCam see-through camera. Our approach utilizes an M. W. Net that integrates a learnable, spatially variant deconvolution layer with a U-Net for fusion and denoising to efficiently solve the complex inverse problem. This method yields a transformative, nearly 4000-fold increase in reconstruction speed while simultaneously providing a significant quality enhancement of almost 8 dB in PSNR compared to previous iterative algorithms. This leap in performance unlocks the potential for real-time, interactive applications for see-through computational cameras.

The practical implementation of this deep learning approach involves several considerations for which clear optimization paths exist. The current need for a large training dataset, for instance, can be significantly mitigated through advanced data augmentation strategies or by leveraging generative models to synthesize diverse training scenarios from a limited number of physical measurements. Similarly, while the current model is specialized for a specific hardware configuration, future work can enhance its flexibility. By employing transfer learning or meta-learning techniques, the network could be rapidly adapted to new optical systems with minimal recalibration data, greatly reducing the retraining overhead. Finally, the present reliance on high-performance GPUs can be addressed through model compression techniques such as quantization and pruning, paving the way for efficient deployment on resource-constrained edge devices and mobile platforms.

Looking ahead, the fusion of advanced computational algorithms with novel optical hardware opens up a vast design space. This framework could be extended beyond 3D imaging to perform more complex real-time tasks, such as glare suppression or hyperspectral analysis, by tailoring the network to different physical priors. These advancements will accelerate the development of imperceptible smart cameras, seamlessly integrating AR into our daily lives.

Acknowledgment. This work was supported by the German Research Foundation (No. CZ 55/48-1), the National Natural Science Foundation of China (No. 62235009), and the National Key Research and Development Program of China (No. 2021YFB2802000).