Photonics Insights, Volume. 4, Issue 2, R03(2025)

Revolutionizing optical imaging: computational imaging via deep learning Story Video , On the Cover

Xiyuan Luo1,†... Sen Wang1, Jinpeng Liu1,2, Xue Dong1, Piao He1, Qingyu Yang1, Xi Chen1, Feiyan Zhou1, Tong Zhang1, Shijie Feng2, Pingli Han1,3, Zhiming Zhou1, Meng Xiang1,3, Jiaming Qian2, Haigang Ma2, Shun Zhou2, Linpeng Lu2, Chao Zuo2,*, Zihan Geng4,*, Yi Wei5,*, and Fei Liu13,* |Show fewer author(s)
Author Affiliations
  • 1School of Optoelectronic Engineering, Xidian University, Xi’an, China
  • 2School of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing, China
  • 3Xi’an Key Laboratory of Computational Imaging, Xi’an, China
  • 4Institute of Data and Information, Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
  • 5Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, USA
  • show less
    References(587)

    [8] N. Li et al. Research on attention-driven unsupervised underwater image enhancement, 1536(2023).

    [11] A. V. Babu et al. Deep learning at the edge enables real-time streaming ptychographic imaging(2022).

    [14] W. X. Zhang et al. Three-dimensional measurement system based on full-field heterodyne interferometry. Opt. Prec. Eng., 27, 2097(2019).

    [15] Y. Liu et al. Research on snapshot infrared computational spectral imaging technology, 11563, 158(2020).

    [34] W. Luo et al. Efficient deep learning for stereo matching, 5695(2016).

    [35] Y. Kuznietsov et al. Semi-supervised deep learning for monocular depth map prediction, 2215(2017).

    [39] R. Swanson et al. Wavefront reconstruction and prediction with convolutional neural networks, 481(2018).

    [40] T. B. DuBose et al. Deep learning for turbulence-tolerant wavefront reconstruction of Shack-Hartmann wavefront sensors, JW4G.6(2020).

    [57] C. Gannon et al. Using machine learning to create high-efficiency freeform illumination design tools(2018).

    [64] A. Nikonorov et al. Deep learning-based imaging using single-lens and multi-aperture diffractive optical systems(2019).

    [65] X. Huang et al. A convolutional neural network based single-frame super-resolution for lensless blood cell counting, 168(2016).

    [66] X. Ma et al. Research on single-frame super-resolution reconstruction algorithm for low resolution cell images based on convolutional neural network, 369(2018).

    [67] C. Dong et al. Accelerating the super-resolution convolutional neural network, 391(2016).

    [69] Y. Fang et al. Classification of white blood cells by convolution neural network in lens-free imaging system, 1(2018).

    [70] S. Li et al. A deep learning feature fusion algorithm based on Lensless cell detection system, 1(2020).

    [84] J. D. Rego et al. Robustness lens image reconstruction via PSF estimation, 403(2021).

    [87] M. S. Asif et al. Flatcam: replacing lenses with masks and computation, 663(2015).

    [90] K. Tajima et al. Lensless light-field imaging with multi-phased fresnel zone aperture, 1(2017).

    [111] R. Shang et al. Deep-learning-driven reliable single-pixel imaging with uncertainty approximation(2021).

    [122] M. Lyu et al. Exploit imaging through opaque wall via deep learning(2017).

    [142] T. Eboli et al. End-to-end interpretable learning of non-blind image deblurring, 314(2020).

    [144] K. Zhang et al. Learning deep CNN denoiser prior for image restoration, 2808(2017).

    [145] J. W. Zhang et al. Learning fully convolutional networks for iterative non-blind deconvolution, 6969(2017).

    [148] L. Chen et al. Learning a non-blind deblurring network for night blurry images, 10542(2021).

    [166] K. Tanaka et al. Polarized non-line-of-sight imaging, 2136(2020).

    [167] C. Zhou et al. Non-line-of-sight imaging off a Phong surface through deep learning(2020).

    [168] R. Ramesh et al. 5d time-light transport matrix: what can we reason about scene properties?. Acta Leprologica, 94, 91411(2018).

    [169] A. B. Yedidia et al. Using unknown occluders to recover hidden scenes, 12231(2019).

    [172] B. Ahn et al. Convolutional approximations to the general non-line-of-sight imaging operator, 7889(2019).

    [180] V. Arellano et al. Fast back-projection for non-line of sight reconstruction, 1(2017).

    [181] D. B. Lindell et al. Acoustic non-line-of-sight imaging, 6780(2019).

    [184] T. Maeda et al. Thermal non-line-of-sight imaging, 1(2019).

    [191] S. W. Seidel et al. Corner occluder computational periscopy: estimating a hidden scene from a single photograph, 1(2019).

    [192] S. I. Young et al. Non-line-of-sight surface reconstruction using the directional light-cone transform, 1407(2020).

    [193] C. Y. Tsai et al. Beyond volumetric albedo--a surface optimization framework for non-line-of-sight imaging, 1545(2019).

    [195] D. Wu et al. Frequency analysis of transient light transport with applications in bare sensor imaging, 542(2012).

    [198] M. Tancik et al. Data-driven non-line-of-sight imaging with a traditional camera, IW2B.6(2018).

    [199] W. Chen et al. Steady-state non-line-of-sight imaging, 6790(2019).

    [201] A. Kirmani et al. Looking around the corner using transient imaging, 159(2009).

    [203] M. Isogawa et al. Optical non-line-of-sight physics-based 3d human pose estimation, 7013(2020).

    [205] J. Grau et al. Occlusion fields: an implicit representation for non-line-of-sight surface reconstruction(2022).

    [215] X. He et al. Passive non-line-of-sight imaging reconstruction based on dual input U-Net, 337(2023).

    [216] X. Yu et al. CAGAN: a channel-aware generative adversarial network for passive non-line-of-sight imaging, 508(2023).

    [218] C. Wang et al. Passive non-line-of-sight imaging of moving targets using physical embedding and event-based vision(2024).

    [227] Y. Y. Schechner et al. Instant dehazing of images using polarization, 1(2001).

    [230] C. Lei et al. Shape from polarization for complex scenes in the wild, 12632(2022).

    [233] Y. Lyu et al. Reflection separation using a pair of unpolarized and polarized images, 32(2019).

    [246] Y. Li et al. Deep learning approach to scalable imaging through scattering media(2019).

    [255] T. Gruber et al. Gated2depth: Real-time dense lidar from gated images, 1506(2019).

    [258] C. Yang et al. Accelerated photoacoustic tomography reconstruction via recurrent inference machines, 6371(2019).

    [264] S. Antholzer et al. Photoacoustic image reconstruction via deep learning, 10494, 433.

    [270] T. Feng et al. Adaptively spatial PSF removal enables contrast enhancement for multi-layer(2024).

    [272] Z. Guan et al. Ptychonet: Fast and High Quality Phase Retrieval for Ptychography(2019).

    [283] F. Shamshad et al. Subsampled Fourier ptychography via pretrained invertible and untrained network priors(2019).

    [284] V. Bianco et al. Deep learning assisted Fourier ptychography for cells and tissue analysis, 12622, 65(2023).

    [286] T. Li et al. Coordinate-based neural network for Fourier phase retrieval, 2585(2024).

    [287] L. Boominathan et al. Phase retrieval for Fourier ptychography under varying amount of measurements(2018).

    [294] S. Gupta et al. Perceptually driven conditional GAN for Fourier ptychography, 1267(2019).

    [295] S. Li et al. Synthetic apertures for array ptychography imaging via deep learning, 12138, 41(2022).

    [296] W. Chen. Fourier ptychography based on a u-net convolutional neural network, 130(2023).

    [305] J. W. Goodman. Introduction to Fourier Optics(1996).

    [306] U. Schanars et al. Digital holography and wavefront sensing: Principles, Techniques and Applications(2014).

    [320] X. Lu et al. Learning video object segmentation from unlabeled videos, 8960(2020).

    [324] H. Altwaijry et al. Learning to detect and match keypoints with deep architectures(2016).

    [328] E. R. Eiríksson et al. Precision and accuracy parameters in structured light 3-D scanning, XL-5/W8, 7(2016).

    [330] Y. R. Guo et al. Method for extracting line structured light center in complex environment. Comput. Eng. Design, 40, 1133(2019).

    [346] J. Beltrán et al. Birdnet: a 3d object detection framework from lidar information, 3517(2018).

    [347] A. Barrera et al. Birdnet+: end-to-end 3d object detection in lidar bird’s eye view, 3517(2020).

    [349] C. R. Qi et al. Pointnet: deep learning on point sets for 3d classification and segmentation, 652(2017).

    [350] M. Tatarchenko et al. Octree generating networks: efficient convolutional architectures for high-resolution 3d outputs, 2088(2017).

    [351] Y. Yao et al. Mvsnet: depth inference for unstructured multi-view stereo, 767(2018).

    [353] G. Xu et al. Attention concatenation volume for accurate and efficient stereo matching, 12981(2022).

    [355] R. Shao et al. Diffustereo: high quality human reconstruction via diffusion-based stereo using sparse cameras, 702(2022).

    [356] D. Zheng et al. Diffuvolume: diffusion model for volume based stereo matching(2023).

    [357] K. Koshikawa. A polarimetric approach to shape understanding of glossy objects. Adv. Robotics, 2, 190(1979).

    [359] R. B. Fisher. From Surfaces to Objects: Computer Vision and Three Dimensional Scene Analysis(1989).

    [361] V. Müller. Elimination of specular surface-reflectance using polarized and unpolarized light, 4, 625(1996).

    [362] Y. Zhao et al. Multi-Band Polarization Imaging(2016).

    [364] Z. Cui et al. Polarimetric multi-view stereo, 1558(2017).

    [365] W. A. P. Smith et al. Linear depth estimation from an uncalibrated, monocular polarisation image, 14, 109(2016).

    [366] A. H. Mahmoud et al. Direct method for shape recovery from polarization and shading, 1769(2012).

    [367] A. Kadambi et al. Polarized 3d: synthesis of polarization and depth cues for enhanced 3d sensing, 1(2015).

    [369] Y. Ba et al. Deep shape from polarization, 16, 554(2020).

    [370] Y. Kondo et al. Accurate polarimetric BRDF for real polarization scene rendering, 16, 220(2020).

    [371] V. Deschaintre et al. Deep polarization imaging for 3d shape and svbrdf acquisition, 15567(2021).

    [372] J. Hur et al. Self-supervised monocular scene flow estimation, 7396(2020).

    [376] T. Huang et al. Learning accurate 3d shape based on stereo polarimetric imaging, 17287(2023).

    [377] R. Girshick et al. Rich feature hierarchies for accurate object detection and semantic segmentation, 580(2014).

    [378] R. Girshick. Fast r-cnn, 1440(2015).

    [380] J. Redmon et al. You only look once: unified, real-time object detection, 779(2016).

    [381] W. Liu et al. SSD: Single shot multibox detector, 14, 21(2016).

    [382] H. Law et al. Cornernet: detecting objects as paired keypoints, 734(2018).

    [384] L. B. Wolff. Surface orientation from polarization images, 850, 110(1988).

    [385] D. H. Goldstein. Polarized Light(2017).

    [386] J. S. Lee et al. Polarimetric Radar Imaging: From Basics to Applications(2017).

    [388] W. Fan et al. Polarization-based car detection, 3069(2018).

    [391] Y. Tian et al. Face anti-spoofing by learning polarization cues in a real-world scenario, 129(2020).

    [395] K. Yang et al. Predicting polarization beyond semantics for wearable robotics, 96(2018).

    [396] Y. Zhang et al. Exploration of deep learning-based multimodal fusion for semantic road scene segmentation(2019).

    [397] M. Blanchon et al. Outdoor scenes pixel-wise semantic segmentation using polarimetry and fully convolutional network(2019).

    [398] Y. Qiao et al. Multi-view spectral polarization propagation for video glass segmentation, 23218(2023).

    [401] H. Farid et al. Separating reflections and lighting using independent components analysis, 262(1999).

    [402] Y. Y. Schechner et al. Polarization-based decorrelation of transparent layers: the inclination angle of an invisible surface, 814(1999).

    [406] P. Wieschollek et al. Separating reflection and transmission images in the wild, 89(2018).

    [407] C. Lei et al. Polarized reflection removal with perfect alignment in the wild, 1750(2020).

    [417] T. Novikova et al. Multi-spectral Mueller matrix imaging polarimetry for studies of human tissues, TTh3B.2(2016).

    [422] Y. Quéau et al. Learning to classify materials using Mueller imaging polarimetry, 11172, 246(2019).

    [424] I. J. Vaughn et al. Classification using active polarimetry, 8364, 243(2012).

    [428] X. Zhou et al. Automatic detection of head and neck squamous cell carcinoma on pathologic slides using polarized hyperspectral imaging and machine learning, 11603, 165(2021).

    [429] S. Mukhopadhyay et al. Optical diagnosis of colon and cervical cancer by support vector machine, 9887, 46(2016).

    [445] C. Hinojosa et al. Spectral-spatial classification from multi-sensor compressive measurements using superpixels, 3413(2019).

    [447] M. Silva-Maldonado et al. End-to-end compressive spectral classification: a deep learning approach applied to the grading of Tahiti lime, 44(2021).

    [448] K. He et al. Deep residual learning for image recognition, 770(2016).

    [455] M. Petrou. Mixed pixel classification: an overview, 69(1999).

    [459] Y. Altmann et al. Supervised nonlinear spectral unmixing using a polynomial post nonlinear model for hyperspectral imagery, 1009(2011).

    [462] J. Plaza et al. Nonlinear mixture models for analyzing laboratory simulated-forest hyperspectral data, 5238, 480(2004).

    [463] J. S. Bhatt et al. Deep learning in hyperspectral unmixing: a review, 2189(2020).

    [465] P. V. Arun et al. Graph neural network based interpretable spectral unmixing for hyperspectral unmixing hyperspectral IIRS data onboard chandrayaan-2 mission, 4202(2023).

    [466] H. Hua et al. A dual-stream convolutional feature fusion network for hyperspectral unmixing, 7531(2023).

    [467] S. K. Bashetti et al. Self-supervised deep network for automatic target recognition in SAR, 8158(2023).

    [471] J. Ma et al. Deep tensor ADMM-net for snapshot compressive imaging, 10223(2019).

    [474] L. Wang et al. Hyperspectral image reconstruction using a deep spatial-spectral prior, 8032(2019).

    [475] L. Wang et al. DNU: deep non-local unrolling for computational spectral imaging, 161(2020).

    [476] D. Ulyanov et al. Deep image prior, 9446(2018).

    [477] D. S. Jeon et al. Compact snapshot hyperspectral imaging with diffracted rotation(2019).

    [480] A. Robles-Kelly. Single image spectral reconstruction for multimedia applications, 251(2015).

    [481] S. Galliani et al. Learned spectral super-resolution(2017).

    [482] S. Nie et al. Deeply learned filter response functions for hyperspectral reconstruction, 4767(2018).

    [484] O. Ronneberger et al. U-net: convolutional networks for biomedical image segmentation, 234(2015).

    [485] W. Shi et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network, 1874(2016).

    [487] D. K. Sahu et al. Different image fusion techniques–a critical review. IJMER, 2, 4298(2012).

    [512] K. Ram Prabhakar et al. Deepfuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs, 4714(2017).

    [528] Y. Liu et al. A medical image fusion method based on convolutional neural networks, 1(2017).

    [532] Y. Zhou et al. Multimodal medical image fusion network based on target information enhancement. IEEE Access, 12, 70851(2024).

    [537] A. Guo et al. Unsupervised blur kernel learning for pansharpening, 633(2020).

    [540] Y. Wu et al. Pansharpening using unsupervised generative adversarial networks with recursive mixed-scale feature fusion. IEEE J. Sel. Topics Appl. Earth Observ. Rem. Sens., 16, 3742(2023).

    [542] L. He et al. Unsupervised pansharpening based on double-cycle consistency(2024).

    [545] X. Mao et al. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections, 26(2016).

    [546] T. Wang et al. Dilated deep residual network for image denoising, 1272(2017).

    [547] J. Guo et al. Toward convolutional blind denoising of real photographs, 1712(2019).

    [551] J. Chen et al. Image blind denoising with generative adversarial network based noise modeling, 3155(2018).

    [553] V. Lempitsky et al. Deep image prior, 9946(2018).

    [554] G. Mataev et al. DeepRED: deep image prior powered by RED(2019).

    [556] J. Lehtinen et al. Noise2Noise: learning image restoration without clean data(2018).

    [557] S. Laine et al. High-quality self-supervised deep image denoising, 32(2019).

    [558] S. Soltanayev et al. Training deep learning based denoisers without ground truth data, 31(2018).

    [559] M. Zhussip et al. Extending Stein’s unbiased risk estimator to train deep denoisers with correlated pairs of noisy images, 32(2019).

    [560] Y. Zhang et al. Kindling the darkness: a practical low-light image enhancer, 1632(2019).

    [561] Y. Xiang et al. WMANet: Wavelet-based multi-scale attention network for low-light image enhancement. IEEE Access, 12, 105674(2024).

    [563] Y. Liu et al. PD-GAN: perceptual-details gan for extremely noisy low light image enhancement, 1840(2021).

    [566] F. Lv et al. MBLLEN: low-light image/video enhancement using CNNS. BMVC, 220, 4(2018).

    [567] J. Perez et al. A deep learning approach for underwater image enhancement, 183(2017).

    [569] J. Li et al. WaterGAN: unsupervised generative network to enable real-time color correction of monocular underwater images. IEEE Rob. Autom. Lett., 3, 387(2017).

    [570] N. Wang et al. UWGAN: underwater GAN for real-world underwater color restoration and dehazing(2019).

    [572] B. Li et al. An all-in-one network for dehazing and beyond(2017).

    [573] W. Ren et al. Gated fusion network for single image dehazing, 3253(2018).

    [577] C. Ledig et al. Photo-realistic single image super-resolution using a generative adversarial network, 4681(2017).

    [580] H. Shen et al. Lossless compression of curated erythrocyte images using deep autoencoders for malaria infection diagnosis, 1(2016).

    [581] F. Mentzer et al. Practical full resolution learned lossless image compression, 10629(2019).

    [582] H. Rhee et al. LC-FDNet: learned lossless image compression with frequency decomposition network, 6033(2022).

    [585] Z. Cheng et al. Learned image compression with discretized gaussian mixture likelihoods and attention modules, 7939(2020).

    [587] A. Tawfik et al. A generic real time autoencoder-based lossy image compression, 1(2022).

    Tools

    Get Citation

    Copy Citation Text

    Xiyuan Luo, Sen Wang, Jinpeng Liu, Xue Dong, Piao He, Qingyu Yang, Xi Chen, Feiyan Zhou, Tong Zhang, Shijie Feng, Pingli Han, Zhiming Zhou, Meng Xiang, Jiaming Qian, Haigang Ma, Shun Zhou, Linpeng Lu, Chao Zuo, Zihan Geng, Yi Wei, Fei Liu, "Revolutionizing optical imaging: computational imaging via deep learning," Photon. Insights 4, R03 (2025)

    Download Citation

    EndNote(RIS)BibTexPlain Text
    Save article for my favorites
    Paper Information

    Category: Review Articles

    Received: Dec. 30, 2024

    Accepted: Mar. 13, 2025

    Published Online: Apr. 9, 2025

    The Author Email: Zuo Chao (zuochao@njust.edu.cn), Geng Zihan (geng.zihan@sz.tsinghua.edu.cn), Wei Yi (yiwei124@mit.edu), Liu Fei (feiliu@xidian.edu.cn)

    DOI:10.3788/PI.2025.R03

    CSTR:32396.14.PI.2025.R03

    Topics