Hyperspectral imaging is employed to capture multi-band spectral images by examining the reflection or radiation data from an object or scene across successive wavelengths of light. With its capacity for high spatial and spectral resolution, hyperspectral imaging has great potential in the applications of biology and medicine [1–3], agriculture and forestry [4–6], oceans and astronomy [7,8], military and defense [9], and art and cultural relics [10]. Traditional hyperspectral imaging systems acquire spectral data through diverse techniques, such as whiskbroom, pushbroom, and wavelength scanning [11–13]. Although yielding accurate spectral components, these systems sacrifice imaging time or space, resulting in the inability to capture dynamic scenes in real time. Thus, researchers have conceived a range of snapshot hyperspectral imaging (SHI) systems to achieve real-time wide-field imaging spectrometers.

SHI consists of an optical hardware encoder and a software decoder. Based on their encoding strategies, contemporary SHI systems can be classified into amplitude-encoding [14–17] and phase-encoding [18–22] categories. The amplitude-encoding SHI systems, typical examples like the coded aperture snapshot spectral imaging (CASSI) system and its variants, consist of front optics, a pseudorandom binary coded aperture, relay lenses, dispersive elements (e.g., grating or prism), and a focal plane array detector [14]. Among them, the pseudorandom binary pattern has a 50%-transmission ratio and is placed at the effective field stop of an imaging system [14,23,24]. While successfully retrieving spectral images through compressed measurements, they fall short in optical throughput and a bulky system [12,25]. In contrast, the phase-encoding methods manipulate the phase of incident light through a custom-designed ultrathin diffractive lens, which yields a coded diffractive image with a spectrum separation [18,21,26]. The phase modulation element in SHI typically involves a diffractive optical element (DOE) [20,27–30], metasurfaces [31,32], and other nanomaterials [33–35]. Among these components, DOE-based SHI stands out due to its simple-to-manufacture, compact, and ultrathin structure, and remarkable dispersion capabilities.

The hardware encoder of DOE-based SHI introduces specific phase delays for different wavelengths by customizing the height map of the DOE. Different patterns can be used to design the height map to achieve phase modulation, such as Fresnel, cubic, multi-focal, diffractive achromat, hybrid diffractive-refractive, and square cubic [36,37]. Related research works have progressively addressed problems such as point spread function (PSF) inhomogeneity at different spectral bands, chromatic aberrations, and mismatches between design and fabrication [18,36,38–40]. Still, non-negligible gaps exist between the ideal physical design and the actual practice of DOE. For instance, limitations in stabilized lithography restrict the quantization levels of DOE height maps to eight levels [39]. Furthermore, when the incident wavelength deviates from the design wavelength, the diffraction efficiency of DOE is significantly reduced [40,41]. In terms of the manufacturing of DOE, the existing deployed DOE configuration in deep optics includes a single DOE [28,39] (conducting imaging and phase modulation) and a DOE (implementing phase modulation) coupled with a simple lens (dedicated to imaging) [42]. As a result, the single DOE configuration is preferred to make the system compact.

The software decoder in the DOE-based SHI is used to solve ill-posed inverse problems to retrieve high-fidelity spectral data cubes from the captured single measurement and various reconstruction methods include analytical-modeling-based methods [18,43] and deep-learning-based methods [12,44–46]. Analytical-modeling-based algorithms adopt handcrafted priors, e.g., total variation and non-local self-similarity, to filter the solution to the desired signal space, which relies on a long iteration time and empirical parameter tuning to get optimum results [43]. The deep-learning-based methods include U-net [47] and its variants [48], RNN [49], Transformer [50], LSTM [17], and Mamba [51]. They have been proposed to achieve end-to-end high-fidelity spectral image reconstruction. However, the neglect of the low-level limitations associated with lithography techniques and the diffraction efficiencies of DOE presents practical challenges in real data reconstruction. These challenges include alignment errors and stray light from the transition area of the ring structure.

Liquid-crystal-on-silicon spatial light modulators (LCoS-SLMs) can dynamically simulate the phase modulation of DOE to generate specific phase delay or optical range difference for different wavelengths by controlling the state of liquid crystal pixels on its surface. The LCoS-SLM supports quantization levels up to 256, allowing floating-point gray-level design schemes [52]. These features can mitigate the low-accuracy issues arising from the height limitation of less than 16th levels in the manufactured DOE. Moreover, the LCoS-SLM can dynamically load multiple DOE simulation patterns with different design wavelengths at a frame rate of 180 Hz. This feature enables higher diffraction efficiency and improves spectral recovery accuracy, meanwhile allowing for portable alteration of the focal length for imaging. Furthermore, the repeatable refresh ability of LCoS-SLM dramatically improves the efficiency and reduces the cost required for manufacturing DOEs, facilitating efficient real-time model debugging in the field. With all these resilient merits, LCoS-SLMs have been employed for phase modulation in achromatic imaging [52], super-resolution imaging [53], ultrafast imaging [54], extended depth of field enhancement [55], and computational holographic imaging [56,57], but little research has been conducted on DOE-based SHI.

To bridge this gap, we propose a new lensless efficient SHI (LESHI) aided by an LCoS-SLM. LESHI utilizes an LCoS-SLM to replace a single fabricated DOE as the hardware encoder, simultaneously realizing imaging and phase modulation. For the software decoding process, we developed a learning algorithm based on the ResU-net architecture, taking into account the sensor’s response function and the diffraction efficiency of the DOE. Using the developed algorithm, high-resolution 31-channel spectral images can be reconstructed from the captured three-channel red-green-blue (RGB) image. To improve diffraction efficiency with a single simulated DOE and explore the switch ability of LCoS-SLM, we propose a distributed diffractive optics (DDO) model by dynamically controlling the light phase. Thus, multiple different phase modulation patterns can be loaded onto the LCoS-SLM, resulting in high reconstruction accuracy and high diffraction efficiency throughout the full visible spectral range (400–700 nm). Furthermore, the LESHI system can realize the modification of the focal length and the field of view without adding other optical components, demonstrating the feature of tunability. The entire imaging system adopts an end-to-end approach to modeling, training, and optimizing, ensuring a high level of integration and coordination to achieve optimal performance. In a nutshell, LESHI not only solves the errors between high-level DOE design and fabrication, as well as the optical alignment difficulties during assembly, but also leverages multiple simulated DOEs for imaging in different spectral bands, thus improving the diffraction efficiency and spectral reconstruction accuracy in the entire visible spectrum. At the same time, it enables convenient modification of focus lengths and real-time on-site debugging, greatly diminishing the production cost and time of DOE. Extensive comprehensive simulations and real-world hardware experiments validate the superior performance of the system.

The schematic of the LESHI system is shown in Fig. 1. A light source (CIE standard illuminant D65, Datacolor Tru-Vue light booth) is used to illuminate the object. The reflected light of the sample passes through the polarizer (GCL-050003), is reflected by a beam splitter (GCC-M402103), and impinges on the LCoS-SLM (FSLM-2K39-P02, 8-bit grayscale level of 256 steps, 180-Hz refresh rate) loaded with optimized DOE patterns. Since the liquid crystal layer has different refractive indices for different wavelengths of the spectrum [52,53], it can produce different phase delays for the entire spectrum like DOE, splitting the continuous hyperspectral data cube. Thus, when a light wave passes through the liquid crystal layer of the LCoS-SLM, the modulation of each pixel causes the phase of the light wave to change. Finally, the phase-modulated light reflected from the LCoS-SLM transmits the beam splitter and is recorded by a color CMOS camera (ME2P-1230-23U3C, which contains a Bayer filter).

The working principle of LESHI is illustrated in Fig. 2(a). In the forward propagation of the model, LESHI sequentially performs the compression of the spectral dataset into a three-channel RGB snapshot, the image reconstruction of the 31-channel spectral cube from the snapshot, and the calculation of the loss function between the reconstruction results and the ground truth. In backward propagation of the model, the model optimizes its variables (e.g., the values of each pixel of the phase modulation pattern and parameters in the neural networks) by minimizing the loss function using the gradient descent methods. Notably, we take the diffraction efficiencies into account in the model, which is missed in existing learning methods [18,22,27–30]. Besides, a rotationally symmetric design [28] was used to reduce the computational complexity of the phase delay pattern.

Figure 2(b) shows the imaging process of the LESHI system with a representative PSF (details in Fig. 7 of Appendix A). A spectral dataset in the visible band with 31 channels and 10-nm spectral resolution convolves the PSF and yields the snapshot. The forward mode of LESHI is expressed as Ic∈{R,G,B}(x,y;λ)=∫λaλb[P(x,y;λ)*I0(x,y;λ)]Rcdλ+η.(1)Here, P(x,y;λ) denotes the PSF. I0(x,y;λ) denotes the spectral image in each channel. * denotes a 2D convolution operator. Rc(λ) denotes the spectral response function of each channel. η is the Gaussian noise. λa and λb are the minimum and maximum wavelengths, respectively. Ic∈{R,G,B} denotes the snapshot (details in Appendix A).

To improve the diffraction efficiency and account for noise effects on the quality of the reconstructed images, the ideal PSF without diffraction efficiency can be transformed into the first-order degenerate PSF (D-PSF). The D-PSF provides a more accurate representation of the imaging system and is expressed as P(x,y;λ)=γ(λ)Pideal+[1−γ(λ)]PBN,(2)where Pideal is the ideal PSF of the diffraction imaging model. PBN denotes Gaussian noise, and the subscript “BN” denotes background noise. γ(λ) denotes diffraction efficiency. The derivation for the diffraction efficiency is shown in Appendix B. Figure 2(c) illustrates the combination of D-PSF, which is described in the DDO model in Appendix C.

After the data acquisition, the captured images are used as input to a customized ResU-net [Fig. 2(d), details in Appendix D], which can retrieve 31-channel spectral images. The loss function L consists of three parts including the reconstruction loss, the diffraction efficiency loss, and the L2 regularization on the network weights: L=1K||I˜−I0||1+α1J||P−Pideal||22+β||W||22.(3)Here, α and β are scale constants set to 10−3 and 10−4, respectively. ||I˜−I0||1 denotes the mean absolute error between the reconstructed hyperspectral image I˜ and the ground truth I0, and K denotes the pixel count of the image. ||P−Pideal||22 denotes the square of norm L2. P denotes D-PSF. J indicates the pixel count of the LCoS-SLM. ||W||22 denotes the square of norm L2, and W denotes the network weights.

To verify the LESHI model, we conducted a comprehensive simulation using the ICVL dataset [58], and it consists of 201 spectral scenes, randomly distributed for training (160 scenes), validation (21 scenes), and testing (20 scenes). To match the hyperparameters of the model, each scene with the size of 1930×1300 was cropped into nine overlapped slices, each of which has a size of 512×512. The scene was placed 1.5 m away from the LCoS-SLM, and the LCoS-SLM was positioned 70 mm away from the sensor plane. The LCoS-SLM had a pixel pitch of 4.5μm×4.5μm, a pixel count of 1024×1024, and an 8-bit gray level of 256. The color CMOS camera had the same pixel count and pitch size as that of the LCoS-SLM.

We conducted a simulation to generate the PSFs by setting the parameters of diffraction imaging and camera spectral response functions. Figure 3(a) shows the ground truth in the test set. We systematically simulated the phase modulation patterns. Specifically, LESHI employs an end-to-end optimization approach to generate a phase modulation pattern loaded onto the LCoS-SLM. The resulting pattern, as shown in Fig. 3(b), is a grayscale pattern with 8-bit and 256-level precision. By adjusting the gray level of each pixel on the liquid crystal, the phase delay magnitudes can be modified across various spectra. The inconsistent diffracted spot sizes in different channels due to the varying degree of phase modulation of the spectra by the system result in a white haze covering the captured RGB images. Figure 3(c) displays the simulated captured images by the color CMOS camera, providing a visual representation of this white haze phenomenon. The customized ResU-net network takes the snapshot as input and reconstructs 31-channel hyperspectral images. Figure 3(d) visually represents the reconstructed spectral image in RGB. Figure 3(e) shows the reconstructed 31-channel spectral images using a single LCoS-SLM loaded with a single simulated DOE, colored with the RGB values of the corresponding wavelengths. In addition, we validated the diffraction efficiency effectiveness of the DDO model.

To verify the accuracy of the models for spectral reconstruction, we compared the average spectral radiance of the reconstructed and true spectral images. Two 4×4-pixel regions [marked by white boxes with numbers “1” and “2” in Fig. 3(a)] were randomly selected in the scene. Their average spectral radiances of reconstruction result and ground truth as a function of wavelength are shown in Figs. 3(f) and 3(g) for regions “1” and “2”, respectively. The difference between the reconstructed images and their ground truth is manifested in the shaded area in Figs. 3(f) and 3(g) with an average value of 3.54% for region “1” and 3.21% for region “2”. LESHI retrieves the changing trend across spectral range well. We compared the diffraction efficiencies of a single DOE pattern [marked by LCOS-S in Fig. 3(h)] and DDO-based DOE patterns [marked by LCOS-D in Fig. 3(h)] at different wavelength bands. The result shows LCoS-D has an average diffraction efficiency of 0.91, greater than 0.75 of LCoS-S. The inserted table in Fig. 3(h) shows the relative diffraction efficiency gain (RDEG) of LCoS-D compared to LCoS-S, quantifying the diffraction efficiency performance improvement rate of LCoS-D at three different bands (400–500 nm, 500–600 nm, and 600–700 nm). To investigate the impact of different levels of DOE patterns on spectral reconstruction, we examined four different levels (i.e., 4, 16, 64, and 256) in LESHI. The corresponding results are shown in Appendix E. Furthermore, we elaborated on the advantages of LCoS-D models over the fabricated DOE and LCoS-S in terms of spectral reconstruction quality in Appendix F. LESHI, when compared to other SHI modalities such as CASSI [14] and Fresnel lenses [36], stands out due to its exceptional performance in compact system design and high spectral reconstruction quality (details are given in Appendix G).

Upon the LESHI model, we built the LESHI system. To characterize the spatial resolution of the LESHI system, the resolution test chart of ISO12233 (3nh, SIQ) was used. The distance between the resolution test chart and the LCoS-SLM was 1.2 m, the screen ratio of the test chart was 4:3, and the focal length of LESHI was set to 50 mm. Moreover, we mitigate the effect of multiple orders by adding a polarizer in front of the LCoS-SLM and increasing its phase quantization level to the highest level (256-level). Figure 4(a) shows the reconstructed resolution test chart, which preserves lots of low- and high-frequency information about the chart. Figures 4(b) and 4(c) plot the reconstructed intensity profiles of the two groups of lines at different locations [marked by light orange and teal boxes in Fig. 4(a)] on the resolution target against the ground truth intensity profiles. With the Rayleigh resolution criterion, the effective spatial resolution of the LESHI system was characterized as 15.74 μm.

The spectral resolution of the LESHI system was evaluated by comparing the spectral values obtained from the spectrometer capturing a ColorChecker Digital SG with the reconstruction results. The measurement of the color calibrator using the LESHI system is shown in Fig. 4(d). Figure 4(e) shows the reconstructed 31-channel spectral composite image in RGB form. Figure 4(f) shows the root mean square error (RMSE, left y-axis) and maximum single-spectrum-channel error (right y-axis) of the real and reconstructed values of the spectral luminance measurement points at six different locations [marked by the white dashed boxes with letters “ABCDEF” in Fig. 4(c)]. Figure 4(g) shows the results of reconstructed values of the spectral luminance measurement points at these six different locations. The reconstruction data demonstrate that the LESHI system’s spectral reconstruction at six distinct locations exhibits minimal error when compared to the actual values. Figure 4(h) showcases the seven representative reconstructed spectral channels of the color calibrator. These 10-nm spectral channels have center wavelengths at 410 nm, 450 nm, 490 nm, 530 nm, 570 nm, 630 nm, and 680 nm. All reconstructed spectral channels are shown in Visualization 1.

To demonstrate the feasibility of applying the DDO model to LESHI, a Thorlabs’ Lab Snacks box is used as the test sample. First, we loaded the three different designed DOE patterns sequentially onto the LCoS-SLM and captured the corresponding RGB images. Second, we extracted the R, G, and B channels from the three captured images. Third, the selected R, G, and B channels with the highest diffraction efficiencies were combined. Finally, the newly synthesized RGB image was fed into the reconstruction network to retrieve 31-channel spectral images. Figure 5(a) shows the measured RGB image using a single DOE pattern (LCoS-S) and the seven representative reconstructed channels (center wavelengths at 410 nm, 450 nm, 490 nm, 530 nm, 570 nm, 630 nm, and 680 nm). Figure 5(b) is the same as Fig. 5(a) except using multiple simulated DOE patterns (LCoS-D). The comparison results show that the DDO-model-based reconstruction results are better than those of the single DOE pattern. All reconstructed spectral images generated by LCoS-S and LCoS-D are shown in Visualization 2.

To quantitively analyze the reconstruction result, we measured the spectral radiance of two local areas [10×10 pixels, marked by white boxes with numbers “1” and “2” in Fig. 5(a)] in the scene using a spectrometer (CS-2000). Figures 5(c) and 5(d) show the comparison results of the spectral radiance between the measured and the reconstructed values of LCoS-S and LCoS-D at the local areas “1” and “2”, respectively. The error (right y-axis) calculated by the subtraction between the reconstructed results and their ground truth is shown as the shaded area in Figs. 5(c) and 5(d). The average error of LCoS-D is 1.84%, smaller than 4.27% of that of LCoS-S. Besides, the reconstruction results show that both LCoS-S and LCoS-D have retrieved the spectral change trend well, compared with the ground truth. LCoS-D has a better performance at 500–700 nm than LCoS-S. This phenomenon indicates that LCoS-S suffers from degradation of reconstruction accuracy due to diffraction inefficiency in some off-center bands, but the LCoS-D model can overcome this issue. The R, G, and B images of the same scene were captured under the LCoS-S [upper panel in Fig. 5(e)] and LCoS-D [bottom panel in Fig. 5(e)] models. The results reveal that the LCoS-D images are more blurred (white haze effect) than those of LCoS-S. This phenomenon can be attributed to the differing diffraction efficiencies across the R, G, and B channels. Higher diffraction efficiencies result in a larger spread of the PSF in each channel, contributing to an increased degree of point scattering in the captured images.

The tunable and convenient focal length of the LESHI system enables it to meet the different needs of the imaging field of view and the range of the captured scene. The focal length of the LESHI system can be modified by loading DOE patterns with different focal lengths. First, we trained the patterns of DOEs with focal lengths ranging from 50 mm to 100 mm, with a step of 2 mm, and a total of 25 images. Six representative DOE patterns (with focal lengths of 50 mm, 60 mm, 70 mm, 80 mm, 90 mm, and 100 mm) are shown in Fig. 6(a). Second, each of the well-trained patterns was loaded on the LCoS-SLM, and the CMOS camera was moved to the corresponding position according to the focal length of the used pattern. Using the captured RGB images [Fig. 6(b)] under different focal lengths as the input of the well-trained neural network, the corresponding reconstructed spectral images can be retrieved with high-fidelity image quality [Fig. 6(c)]. The results show that the field of view of the scene shrinks as the focal length increases, which can be explained by the Lagrange-Helmholtz invariant (i.e., a bigger focal length gives a smaller aperture angle in image space and thus a smaller object height). Figure 6(d) shows one representative reconstructed spectral image under these focal lengths. The reconstructed spectral images under the focal length range (50–100 mm) are shown in Visualization 3. Notably, no more optical elements are modified or added in the LESHI system during focal length changes, which dramatically reduces the complexity of the zoom optical system.

We have developed the LESHI system based on diffractive optics via the LCoS-SLM. LESHI employs a learning-based DOE pattern loaded onto the LCoS-SLM to perform phase modulation and imaging, instead of a physically fabricated DOE. Using the customized ResU-net algorithm, we have retrieved the 31-channel spectral cube with an image resolution of 1920×1080 pixels, an effective spatial resolution of 41.74 μm, and a spectral resolution of 10 nm across 400–700 nm from the color image captured by the CMOS camera. The comprehensive process of wavefront modulation, imaging, and spectral reconstruction is achieved through an end-to-end design approach. This approach combines a diffraction imaging optical model with a deep-learning-based reconstruction algorithm. By doing so, it optimizes the appropriate phase modulation profile. We have proposed the DDO imaging model that utilizes the dynamic refresh ability of LCoS-SLM to load multiple DOE patterns applicable to different bands for specific phase modulation and imaging. This model improves average diffraction efficiency from 0.75 to 0.91 across the entire spectral band. Meanwhile, the diffraction efficiency is considered in the PSF model to generate a new D-PSF to optimize the PSF in real scenes. Furthermore, the LESHI system has a real-time zoom function. We can load the trained patterns with different focal lengths onto the LCoS-SLM to modify the focal length and field of view without adding other optical components. The focal length and field of view are tunable in the model training. Many simulations and practical experiments demonstrate the superiority of the method in spectral image reconstruction.

Compared to diffractive hyperspectral imaging via a fabricated DOE, the LESHI system has significant advantages in terms of spectral reconstruction accuracy, system flexibility, diffraction efficiency, and cost of fabrication. The limitation of stabilized lithography technology restricts the number of quantization levels supported by fabricated DOE to only eight. This reduction in quantization levels results in a decrease in the resolution of spectral phase modulation by DOE. Consequently, the accuracy of the reconstruction capability of the entire system is weakened. The LCoS-SLM technology offers a phase modulation level of 256 gray levels, allowing for a floating-point gray level design. This feature enables higher phase resolution, which is beneficial for optimizing and replacing fabricated DOE. The high diffraction efficiency of the fabricated DOE is challenging to maintain across the entire 400–700 nm band due to limitations in material and design wavelength. The LESHI system employs the DDO model to dynamically load multiple phase modulation patterns for different spectral bands. This implementation enhances the diffraction efficiency of imaging. Besides, the high cost of DOE fabrication significantly restricts its potential applications. By dynamically loading patterns, LCoS-SLM can save the time and cost of fabricating DOEs, improving the efficiency of real-time system debugging. In addition, the micrometer-scale level presents practical challenges when attempting to achieve pixel-level alignment for DOE. This can result in calibration errors between the idealized camera model and the actual experiment. In contrast, the pattern loaded on the LCoS-SLM has pixel-level translation, rotation, and grayscale flipping, which mitigates the difficulty of optical alignment in practical assembly.

The principle of LESHI could be extended to other DOE-based imaging modalities. The LCoS-SLM can simulate DOE based on various patterns using high-level encoding and reloadable features, thereby improving the performance and efficiency of existing fabricated-DOE-based systems such as full-spectrum computational imaging [18], high-dynamic-range imaging [30], depth-spectral imaging [27], and achromatic extended depth of field and super-resolution imaging [36]. Besides, with an ultrashort chirped pulse as a light source, LESHI could be directly applied to ultrafast imaging [59] because the reconstructed spectral frames of LESHI can be linked to time information benefiting from the chirped pulse (i.e., the wavelength changes during the duration of the pulse).

While the proposed distributed LESHI system improves the spectral imaging performance of the scene, the current model is based on the training of one dataset, which limits its ability to generalize to scenes in wide applications. In the future, the system will be comprehensively optimized by adding the required scene object information to the model training to improve the generalization ability of the model. In addition, deep unfolding networks [60] and plug-and-play mechanisms [61] will be considered to improve the flexibility of the network structure in handling different sizes of spectral cubes. Finally, the entire network model can be miniaturized by optimizing the network parameters, and the trained model can be loaded using FPGA hardware instead of GPU to improve the reconstruction speed of the spectrum.

Acknowledgment. The data table of LCoS-SLM spectral phase delay at different center wavelengths was provided by Xi’an CAS Microstar Optoelectronic Technology Co., Ltd.