Fig. 1. Schematic of the lensless efficient snapshot hyperspectral imaging (LESHI) system. LCoS-SLM, liquid crystal on silicon-based spatial light modulator. LESHI comprises hardware-based diffractive imaging and software-based hyperspectral reconstruction algorithms. The diffractive imaging component includes an LCoS-SLM, a polarizer, a beam splitter, and a color CMOS camera. The hyperspectral reconstruction algorithm employs a ResU-net to decode the spectral information.

Fig. 2. Working principle of LESHI. (a) Pipeline of LESHI. nz denotes the number of spectral channels from λ0 to λn. η denotes sensor noise. * denotes the convolution operator. ∂∂P and ∂∂yin denote the derivative of the imaging model with respect to PSF and the derivative of the reconstructed network with respect to the captured image, respectively. Lh and Lde denote loss of hyperspectral image reconstruction and loss of diffraction efficiency, respectively. ||W||22 denotes the square of norm L2 and W denotes the network weights; β are scale constants set to 10−4. (b) Schematic of PSF acquisition process in diffractive optical imaging based on LCoS-SLM with DOE patterns. I0(x,y;λ) denotes input scene and Ic(x,y;λ) is its convolution result with PSF, P(x,y;λ). (c) DDO model design based on LCoS-SLM. DDO fuses the PSFs of individual DOEs of the different bands and adds the model of the diffraction efficiency to form a degenerate PSF model. (d) Structure of the ResU-net reconstruction algorithm, which combines the U-shaped architecture of U-net with the residual connections of ResNet.

Fig. 3. Validation of LESHI model. (a) Ground truth from the ICVL dataset. (b) The trained simulated DOE pattern loaded on the LCoS-SLM. (c) RGB image generated by the LESHI model with a single DOE pattern. (d) Reconstructed result of (c). (e) Reconstructed hyperspectral images using LESHI model with a single DOE pattern. (f) Ground truth and reconstructed values of the spectral radiance curves for local area “1” marked in (a). (g) Same as (f) but for local area “2”. (h) Diffraction efficiency as a function of wavelength, using single DOE pattern (LCoS-S) and multiple DOE patterns (LCoS-D) in the LESHI model. The table shows the relative diffraction efficiency gain (RDEG) of LCoS-D compared to LCoS-S at three different bands (400–500 nm, 500–600 nm, 600–700 nm).

Fig. 4. Characterization of the LESHI system performance. (a) Reconstructed image of ISO12233 test chart. (b) Spatial line profiles of two regions on the test chart, highlighted in light orange and teal boxes at the location of label 1 in (a). (c) Spatial line profiles of two regions on the test chart, highlighted in light blue and teal boxes at the location of label 2 in (a). (d) Measurement of the LEHSI system. (e) Reconstruction result of (c) in RGB format. (f) Root mean square error (RMSE) and maximum error of reconstructed image and measurement by the CS-2000 spectrometer at six local regions [marked by white boxes in (c)]. (g) Reconstruction radiance curves of six local regions [marked by white boxes in (c)] as a function of wavelength. Ground truth is obtained by the CS-2000 spectrometer. (h) Seven representative reconstructed spectral channels of (d).

Fig. 5. Demonstration of distributed diffractive optics (DDO) imaging. (a) Captured and reconstructed images based on a single simulation of DOE. (b) Captured and reconstruction images based on multiple simulated DOEs (DDO model). (c) Reconstructed values and ground truth of spectral radiance based on LCoS-S and LCoS-D models at the location of label 1 in (a). (d) Reconstructed values and ground truth of spectral radiance based on LCoS-S and LCoS-D models at the location of label 2 in (a). (e) Images and simulated diffraction efficiency (DE) of the R, G, and B channels captured by the model based on LCoS-S and LCoS-D.

Fig. 6. Application results for focal length modification. (a) Phase modulation patterns loaded onto LCoS-SLM with different focal lengths by end-to-end training. (b) Corresponding captured RGB images of (a). (c) Results of spectral image recovery by applying the LESHI system at different focal lengths. (d) Six representative reconstructed spectral channels corresponding to (c).

Fig. 7. LESHI-based point spread function for 31 channels at 400–700 nm. Due to the phase delay of LCoS-SLM for different spectra, the system has different point spread functions for different bands.

Fig. 8. Spectral response and modulation simulation curves of camera and LCoS-SLM. (a) Sensor spectral response curves. (b) Phase modulation curves of LCoS-SLM with different center wavelengths. (c) Diffraction efficiency of LCoS-SLM with different center wavelengths.

Fig. 9. The effect of different levels of the simulated DOE for spectral reconstruction. Comparing the reconstruction performance for 4, 16, 64, and 256 levels, it can be concluded that the reconstruction performance gradually improves with the growth of levels.

Fig. 10. Comparison of spectral reconstruction simulations for different models. (a) Comparing the four reconstruction data results and visual effects, the diffractive optical imaging model based on LCoS-SLM can effectively improve the reconstruction performance and avoid the degradation of the reconstruction results caused by the quantized DOE. (b) Spectral radiance curves for different models. The spectral curves show that the reconstructed spectral curves of LCoS-D are closer to the ground truth values.

Fig. 11. Performance comparison of hyperspectral reconstruction using fabricated DOE and simulated DOE loaded onto LCoS-SLM. (a) Comparison of PSNR for hyperspectral image reconstruction with different models. (b) Comparison of SSIM metrics for hyperspectral image reconstruction with different models. (c) Comparison of RMSE metrics for hyperspectral image reconstruction with different models. (d) Comparison of ERGAS metrics for hyperspectral image reconstruction with different models.