Laser & Optoelectronics Progress, Volume. 61, Issue 2, 0211030(2024)

Review of Optical Pre-Sensor Computing Technology and Its Satellite Remote Sensing Applications (Invited)

Tianyu Li1, Guoqing Wang1、*, Wei Li2、**, Hongwei Chen3、***, Xun Liu2, Zhibin Wang1, Shaochong Liu1, and Yanxin Cai2
Author Affiliations
  • 1School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, Sichuan , China
  • 2Beijing Institute of Space Mechanics and Electricity, Beijing 100094, China
  • 3Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
  • show less
    References(110)

    [1] Huang Z H, Fang H, Li Q et al. Optical remote sensing image enhancement with weak structure preservation via spatially adaptive gamma correction[J]. Infrared Physics & Technology, 94, 38-47(2018).

    [2] Ye B, Tian S F, Ge J et al. Assessment of WorldView-3 data for lithological mapping[J]. Remote Sensing, 9, 1132(2017).

    [3] Pan T. Technical characteristics of Gaofen-2 satellite[J]. Aerospace China, 3-9(2015).

    [4] Li W, Liu X, Zhang W C et al. The application of deep learning in space-based intelligent optical remote sensing[J]. Spacecraft Recovery & Remote Sensing, 41, 56-65(2020).

    [5] Shastri B J, Tait A N, de Lima T F et al. Photonics for artificial intelligence and neuromorphic computing[J]. Nature Photonics, 15, 102-114(2021).

    [6] Fu T Z, Sun R, Huang Y Y et al. Review of on-chip integrated optical neural networks[J]. Chinese Journal of Lasers, 51, 0119001(2024).

    [7] Lin X, Rivenson Y, Yardimci N T et al. All-optical machine learning using diffractive deep neural networks[J]. Science, 361, 1004-1008(2018).

    [8] Chen B, Zhang Z Y, Dai T G et al. Photonic neural networks and its applications[J]. Laser & Optoelectronics Progress, 60, 0600001(2023).

    [9] Hossain M S, Shahriar G M, Mahbubul Syeed M M et al. Region of interest (ROI) selection using vision transformer for automatic analysis using whole slide images[J]. Scientific Reports, 13, 11314(2023).

    [10] Zhou Y, Li R Z, Yu X H et al. Progress in study and application of optical field modulation technology based on liquid crystal spatial light modulators (invited)[J]. Acta Photonica Sinica, 50, 1123001(2021).

    [11] Dudley D, Duncan W M, Slaughter J. Emerging digital micromirror device (DMD) applications[J]. Proceedings of SPIE, 4985, 14-25(2003).

    [12] Wang Y J, Chen H Z, Liu Y Y et al. Application of DMD to high dynamic scene image detection system[J]. Optics and Precision Engineering, 22, 2508-2517(2014).

    [13] Luo H. Research on pixel-level dimming technology based on digital micromirror device[D](2020).

    [14] Yang C. Optimization of DMD lithography imaging quality based on intelligent algorithm[D](2023).

    [15] Sun Y Q, Hu Y, Wang Y Q et al. Analysis on aberration of digital micromirror device in convergent imaging path[J]. Acta Optica Sinica, 39, 0311001(2019).

    [16] Li Y L, Fu X L[M]. Spatial light modulator and its application(1996).

    [17] Chen H. Research on multimode fiber imaging based on phase modulation and deep learning[D](2022).

    [18] Zeng Z. Research on key technology of optical surface shape measurement based on spatial light modulator[D](2023).

    [19] Poleshchuk A G. Fabrication and application of diffractive optical elements[J]. Proceedings of SPIE, 7544, 75443L(2010).

    [20] Danziger Y, Hasman E, Friesem A A et al. Multilevel diffractive elements for generalized wavefront shaping[J]. Optical Engineering, 35, 2556-2565(1996).

    [21] Hu H Q, Zhou H, Xu Z H et al. Practical snapshot hyperspectral imaging with DOE[J]. Optics and Lasers in Engineering, 156, 107098(2022).

    [22] Ryf R, Stuart H R, Giles C R. MEMS tip/tilt and piston mirror arrays as diffractive optical elements[J]. Proceedings of SPIE, 5894, 58940C(2005).

    [23] Yang L L, Cui Q F, Liu T et al. Effects of manufacturing errors on diffraction efficiency for multilayer diffractive optical elements[J]. Applied Optics, 50, 6128-6133(2011).

    [24] Kwon H, Arbabi E, Kamali S M et al. Single-shot quantitative phase gradient microscopy using a system of multifunctional metasurfaces[J]. Nature Photonics, 14, 109-114(2020).

    [25] Xiong B, Liu Y, Xu Y H et al. Breaking the limitation of polarization multiplexing in optical metasurfaces with engineered noise[J]. Science, 379, 294-299(2023).

    [26] Khorasaninejad M, Chen W T, Devlin R et al. Metalenses at visible wavelengths: diffraction-limited focusing and subwavelength resolution imaging[J]. Science, 352, 1190-1194(2016).

    [27] Levanon N, Indukuri S R K C, Frydendahl C et al. Angular transmission response of In-plane symmetry-breaking quasi-BIC all-dielectric metasurfaces[J]. ACS Photonics, 9, 3642-3648(2022).

    [29] Guo C, Xiao M, Minkov M et al. Photonic crystal slab Laplace operator for image differentiation[J]. Optica, 5, 251-256(2018).

    [30] Cordaro A, Kwon H, Sounas D et al. High-index dielectric metasurfaces performing mathematical operations[J]. Nano Letters, 19, 8418-8423(2019).

    [31] Zhou Y, Zheng H Y, Kravchenko I I et al. Flat optics for image differentiation[J]. Nature Photonics, 14, 316-323(2020).

    [32] Fu W W, Zhao D, Li Z Q et al. Ultracompact meta-imagers for arbitrary all-optical convolution[J]. Light: Science & Applications, 11, 62(2022).

    [33] Wang H W, Guo C, Zhao Z X et al. Compact incoherent image differentiation with nanophotonic structures[J]. ACS Photonics, 7, 338-343(2020).

    [34] Zhang X M, Bai B F, Sun H B et al. Incoherent optoelectronic differentiation based on optimized multilayer films[J]. Laser & Photonics Reviews, 16, 2200038(2022).

    [35] Zheng H Y, Liu Q, Zhou Y et al. Meta-optic accelerators for object classifiers[J]. Science Advances, 8, eabo6410(2022).

    [36] Bernstein L, Sludds A, Panuski C et al. Single-shot optical neural network[J]. Science Advances, 9, eadg7904(2023).

    [38] Wang X, Ma X, Ke J et al. Advances in speckle and compressive computational imaging[J]. Acta Optica Sinica, 43, 1511001(2023).

    [39] Nyquist H. Certain topics in telegraph transmission theory[J]. Transactions of the American Institute of Electrical Engineers, 47, 617-644(1928).

    [40] Gan L. Block compressed sensing of natural images[C], 403-406(2007).

    [41] Yuan X, Brady D J, Katsaggelos A K. Snapshot compressive imaging: theory, algorithms, and applications[J]. IEEE Signal Processing Magazine, 38, 65-88(2021).

    [42] Wagadarikar A A, Pitsianis N P, Sun X B et al. Spectral image estimation for coded aperture snapshot spectral imagers[J]. Proceedings of SPIE, 7076, 707602(2008).

    [43] Gehm M E, John R, Brady D J et al. Single-shot compressive spectral imaging with a dual-disperser architecture[J]. Optics Express, 15, 14013-14027(2007).

    [44] Wagadarikar A, John R, Willett R et al. Single disperser design for coded aperture snapshot spectral imaging[J]. Applied Optics, 47, B44-B51(2008).

    [45] Veeraraghavan A, Reddy D, Raskar R. Coded strobing photography: compressive sensing of high speed periodic videos[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33, 671-686(2011).

    [46] Holloway J, Sankaranarayanan A C, Veeraraghavan A et al. Flutter Shutter Video Camera for compressive sensing of videos[C](2012).

    [47] Reddy D, Veeraraghavan A, Chellappa R. P2C2: Programmable pixel compressive camera for high speed imaging[C], 329-336(2011).

    [48] Llull P, Liao X J, Yuan X et al. Coded aperture compressive temporal imaging[J]. Optics Express, 21, 10526-10545(2013).

    [49] Koller R, Schmid L, Matsuda N et al. High spatio-temporal resolution video with compressed sensing[J]. Optics Express, 23, 15992-16007(2015).

    [50] Yang P H, Kong L H, Liu X Y et al. Shearlet enhanced snapshot compressive imaging[J]. IEEE Transactions on Image Processing, 29, 6466-6481(2020).

    [51] Liu Y, Yuan X, Suo J L et al. Rank minimization for snapshot compressive imaging[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41, 2990-3006(2019).

    [52] Boyd P S, Parikh N, Chu E et al. Distributed optimization and statistical learning via the alternating direction method of multipliers[J]. Foundations and Trends® in Machine Learning, 3, 128(2010).

    [53] Yuan X, Jiang H, Huang G et al. SLOPE: shrinkage of local overlapping patches estimator for lensless compressive imaging[J]. IEEE Sensors Journal, 16, 8091-8102(2016).

    [54] Metzler C A, Maleki A, Baraniuk R G. From denoising to compressed sensing[J]. IEEE Transactions on Information Theory, 62, 5117-5144(2016).

    [55] Yuan X, Llull P, Liao X J et al. Low-cost compressive sensing for color video and depth[C], 3318-3325(2014).

    [56] Yuan X. Generalized alternating projection based total variation minimization for compressive sensing[C], 2539-2543(2016).

    [57] Yang J B, Liao X J, Yuan X et al. Compressive sensing by learning a Gaussian mixture model from measurements[J]. IEEE Transactions on Image Processing, 24, 106-119(2015).

    [58] Yang J B, Yuan X, Liao X J et al. Video compressive sensing using Gaussian mixture models[J]. IEEE Transactions on Image Processing, 23, 4863-4878(2014).

    [59] Renna F, Wang L M, Yuan X et al. Classification and reconstruction of high-dimensional signals from low-dimensional features in the presence of side information[J]. IEEE Transactions on Information Theory, 62, 6459-6492(2016).

    [60] Yuan X, Tsai T H, Zhu R Y et al. Compressive hyperspectral imaging with side information[J]. IEEE Journal of Selected Topics in Signal Processing, 9, 964-976(2015).

    [61] Qiao M, Meng Z Y, Ma J W et al. Deep learning for video compressive sensing[J]. APL Photonics, 5, 030801(2020).

    [62] Zhang B, Yuan X, Deng C et al. End-to-end snapshot compressed super-resolution imaging with deep optics[J]. Optica, 9, 451-454(2022).

    [65] Cheng Z H, Lu R Y, Wang Z J, Vedaldi A, Bischof H, Brox T et al. BIRNAT: bidirectional recurrent neural networks with adversarial training for video snapshot compressive imaging[M]. Computer vision-ECCV 2020. Lecture notes in computer science, 12369, 258-275(2020).

    [66] Cheng Z H, Chen B, Liu G L et al. Memory-efficient network for large-scale video compressive sensing[C], 16241-16250(2021).

    [67] Li L G, Wang L Z, Song W T et al. Quantization-aware deep optics for diffractive snapshot hyperspectral imaging[C], 19748-19757(2022).

    [68] Hu X W, Cai Y H, Lin J et al. HDNet: high-resolution dual-domain learning for spectral compressive imaging[C], 17521-17530(2022).

    [69] Meng Z Y, Ma J W, Yuan X, Vedaldi A, Bischof H, Brox T et al. End-to-end low cost compressive spectral imaging with spatial-spectral self-attention[M]. Computer vision-ECCV 2020. Lecture notes in computer science, 12368, 187-204(2020).

    [70] Cai Y H, Lin J, Hu X W et al. Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction[C], 17481-17490(2022).

    [71] Cai Y H, Lin J, Hu X W, Avidan S, Brostow G, Cissé M et al. Coarse-to-fine sparse transformer for hyperspectral image reconstruction[M]. Computer vision-ECCV 2022. Lecture notes in computer science, 13677, 686-704(2022).

    [76] Yang Y, Sun J, Li H B et al. Deep ADMM-Net for compressive sensing MRI[C], 10-18(2016).

    [77] Ma J W, Liu X Y, Shou Z et al. Deep tensor ADMM-net for snapshot compressive imaging[C], 10222-10231(2020).

    [78] Yuan X, Liu Y, Suo J L et al. Plug-and-play algorithms for large-scale snapshot compressive imaging[C], 1444-1454(2020).

    [79] Wu Z, Zhang J, Mou C. Dense deep unfolding network with 3D-CNN prior for snapshot compressive imaging[C], 4872-4881(2022).

    [80] Wang L Z, Sun C, Zhang M Q et al. DNU: deep non-local unrolling for computational spectral imaging[C], 1658-1668(2020).

    [81] Wang L Z, Sun C, Fu Y et al. Hyperspectral image reconstruction using a deep spatial-spectral prior[C], 8024-8033(2020).

    [82] Huang T, Dong W S, Yuan X et al. Deep Gaussian scale mixture prior for spectral compressive imaging[C], 16211-16220(2021).

    [84] Wang P, Yuan X. SAUNet: spatial-attention unfolding network for image compressive sensing[C], 5099-5108(2023).

    [85] Wu Z L, Yang C S, Su X F et al. Adaptive deep PnP algorithm for video snapshot compressive imaging[J]. International Journal of Computer Vision, 131, 1662-1679(2023).

    [86] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 60, 84-90(2017).

    [87] He K M, Zhang X Y, Ren S Q et al. Deep residual learning for image recognition[C], 770-778(2016).

    [88] Vaswani A, Shazeer N, Parmar N et al. Attention is all you need[C], 6000-6010(2017).

    [89] Goodfellow I, Pouget-Abadie J, Mirza M et al. Generative adversarial networks[J]. Communications of the ACM, 63, 139-144(2020).

    [90] Zuo Y, Li B H, Zhao Y J et al. All-optical neural network with nonlinear activation functions[J]. Optica, 6, 1132-1137(2019).

    [91] Bueno J, Maktoobi S, Froehly L et al. Reinforcement learning in a large-scale photonic recurrent neural network[J]. Optica, 5, 756-760(2018).

    [92] Li J X, Mengu D, Luo Y et al. Class-specific differential detection in diffractive optical neural networks improves inference accuracy[J]. Advanced Photonics, 1, 046001(2019).

    [93] Yan T, Wu J M, Zhou T K et al. Fourier-space diffractive deep neural network[J]. Physical Review Letters, 123, 023901(2019).

    [94] Shen Y C, Harris N C, Skirlo S et al. Deep learning with coherent nanophotonic circuits[J]. Nature Photonics, 11, 441-446(2017).

    [95] Zhu H H, Zou J, Zhang H et al. Space-efficient optical computing with an integrated chip diffractive neural network[J]. Nature Communications, 13, 1044(2022).

    [96] Fu T Z, Zang Y B, Huang Y Y et al. Photonic machine learning with on-chip diffractive optics[J]. Nature Communications, 14, 70(2023).

    [97] Cui T J, Qi M Q, Wan X et al. Coding metamaterials, digital metamaterials and programmable metamaterials[J]. Light: Science & Applications, 3, e218(2014).

    [98] Liu C, Ma Q, Luo Z J et al. A programmable diffractive deep neural network based on a digital-coding metasurface array[J]. Nature Electronics, 5, 113-122(2022).

    [99] Williamson I A D, Hughes T W, Minkov M et al. Reprogrammable electro-optic nonlinear activation functions for optical neural networks[J]. IEEE Journal of Selected Topics in Quantum Electronics, 26, 7700412(2020).

    [100] Vandoorne K, Dierckx W, Schrauwen B et al. Toward optical signal processing using photonic reservoir computing[J]. Optics Express, 16, 11182-11192(2008).

    [101] Vandoorne K, Dambre J, Verstraeten D et al. Parallel reservoir computing using optical amplifiers[J]. IEEE Transactions on Neural Networks, 22, 1469-1481(2011).

    [102] Mesaritakis C, Papataxiarhis V, Syvridis D. Micro ring resonators as building blocks for an all-optical high-speed reservoir-computing bit-pattern-recognition system[J]. Journal of the Optical Society of America B, 30, 3048-3055(2013).

    [103] Rosenbluth D, Kravtsov K, Fok M P et al. A high performance photonic pulse processing device[J]. Optics Express, 17, 22767-22772(2009).

    [104] Li W, Liu X, Zhang W C et al. Space based edge intelligent remote sensing[J]. Spacecraft Recovery & Remote Sensing, 43, 1-11(2022).

    [105] Zhang F, Chen X Q, Yu S et al. On-orbit artificial intelligence based on space edge computing[J]. Aerospace Shanghai (Chinese & English), 38, 19-24(2021).

    [106] Gunasheela K S, Prasantha H S, Guru D S, Vasudev T, Chethan et al. Satellite image compression-detailed survey of the algorithms[M]. Proceedings of international conference on cognition and recognition. Lecture notes in computer science, 14, 187-198(2018).

    [107] Hussain A J, Al-Fayadh A, Radi N. Image compression techniques: a survey in lossless and lossy algorithms[J]. Neurocomputing, 300, 44-69(2018).

    [108] Liu S Y, Deng W H. Very deep convolutional neural network based image classification using small training sample size[C], 730-734(2016).

    [110] Qian G W, Xu X Q, Zhu S et al. Quantum induced coherence light detection and ranging[J]. Physical Review Letters, 131, 033603(2023).

    Tools

    Get Citation

    Copy Citation Text

    Tianyu Li, Guoqing Wang, Wei Li, Hongwei Chen, Xun Liu, Zhibin Wang, Shaochong Liu, Yanxin Cai. Review of Optical Pre-Sensor Computing Technology and Its Satellite Remote Sensing Applications (Invited)[J]. Laser & Optoelectronics Progress, 2024, 61(2): 0211030

    Download Citation

    EndNote(RIS)BibTexPlain Text
    Save article for my favorites
    Paper Information

    Category: Imaging Systems

    Received: Nov. 15, 2023

    Accepted: Dec. 13, 2023

    Published Online: Feb. 6, 2024

    The Author Email: Wang Guoqing (gqwang0420@uestc.edu.cn), Li Wei (wei_li_bj@163.com), Chen Hongwei (chenhw@tsinghua.edu.cn)

    DOI:10.3788/LOP232509

    Topics