Advances in Deep Learning for Super-Resolution Microscopy（Invited）

Xinyi Lu; Yu Huang; Zitong Zhang; Tianxiao Wu; Hongjun Wu; Yongtao Liu; Zhong Fang; Chao Zuo; Qian Chen

doi:10.3788/LOP241455

Laser & Optoelectronics Progress, Volume. 61, Issue 16, 1611002(2024)

Advances in Deep Learning for Super-Resolution Microscopy（Invited）

Xinyi Lu^1,2, Yu Huang³, Zitong Zhang⁴, Tianxiao Wu^1,2, Hongjun Wu^1,2, Yongtao Liu^1,2、*, Zhong Fang^3、**, Chao Zuo^1,2、***, and Qian Chen^1,2

Author Affiliations

¹Smart Computational Imaging Laboratory, College of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing 210094, Jiangsu, China

²Key Laboratory of Spectral Imaging & Intelligent Sense, Nanjing University of Science and Technology, Nanjing 210094, Jiangsu, China

³School of Mechanical Engineering, Nanjing University of Science and Technology, Nanjing 210094, Jiangsu, China

⁴Infection Management Department of Shenzhen Sami Medical Center (Shenzhen Fourth People's Hospital), Shenzhen 518118, Guangdong, China

show less

Abstract Get PDF(in Chinese)

References(74)

[1] Chen X D, Zou C L, Gong Z J et al. Subdiffraction optical manipulation of the charge state of nitrogen vacancy center in diamond[J]. Light: Science & Applications, 4, e230(2015).

[2] Yang T J, Luo Y R, Ji W et al. Advancing biological super-resolution microscopy through deep learning: a brief review[J]. Biophysics Reports, 7, 253-266(2021).

[3] Rust M J, Bates M, Zhuang X W. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM)[J]. Nature Methods, 3, 793-795(2006).

[4] Betzig E, Patterson G H, Sougrat R et al. Imaging intracellular fluorescent proteins at nanometer resolution[J]. Science, 313, 1642-1645(2006).

[5] Chen C H, Wang F, Wen S H et al. Multi-photon near-infrared emission saturation nanoscopy using upconversion nanoparticles[J]. Nature Communications, 9, 3290(2018).

[6] Denk W, Strickler J H, Webb W W. Two-photon laser scanning fluorescence microscopy[J]. Science, 248, 73-76(1990).

[7] Liu Y T, Wang F, Lu H X et al. Super-resolution mapping of single nanoparticles inside tumor spheroids[J]. Small, 16, e1905572(2020).

[8] Liu Y T, Wen S H, Wang F et al. Population control of upconversion energy transfer for stimulation emission depletion nanoscopy[J]. Advanced Science, 10, e2205990(2023).

[9] Cao Y Y, Li X P, Gu M. Super-resolution nanofabrication with metal-ion doped hybrid material through an optical dual-beam approach[J]. Applied Physics Letters, 105, 263102(2014).

[10] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 60, 84-90(2017).

[11] Friedrich M, Gan Q, Ermolayev V et al. STED-SPIM: stimulated emission depletion improves sheet illumination microscopy resolution[J]. Biophysical Journal, 100, L43-L45(2011).

[12] Ronneberger O, Fischer P, Brox T, Navab N, Hornegger J, Wells W M et al. U-Net: convolutional networks for biomedical image segmentation[M]. Medical image computing and computer-assisted intervention- MICCAI 2015. Lecture notes in computer science, 9351, 234-241(2015).

[13] He K M, Zhang X Y, Ren S Q et al. Deep residual learning for image recognition[C], 770-778(2016).

[14] Ren S Q, He K M, Girshick R et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 1137-1149(2017).

[15] Wang J L, Yan W, Zhang J et al. New advances in the research of stimulated emission depletion super-resolution microscopy[J]. Acta Physica Sinica, 69, 108702(2020).

[16] Schermelleh L, Ferrand A, Huser T et al. Super-resolution microscopy demystified[J]. Nature Cell Biology, 21, 72-84(2019).

[17] Ebrahimi V, Stephan T, Kim J et al. Deep learning enables fast, gentle STED microscopy[J]. Communications Biology, 6, 674(2023).

[18] Li M Z. Deep adversarial network for super stimulated emission depletion imaging[J]. Journal of Nanophotonics, 14, 016009(2020).

[19] An S, Dan D, Yu X H et al. Progress and prospect of research on single-molecule localization super-resolution microscopy (invited review)[J]. Acta Photonica Sinica, 49, 0918001(2020).

[20] Nehme E, Weiss L E, Michaeli T et al. Deep-STORM: super-resolution single-molecule microscopy by deep learning[J]. Optica, 5, 458-464(2018).

[21] Gaire S K, Zhang Y, Li H Y et al. Accelerating multicolor spectroscopic single-molecule localization microscopy using deep learning[J]. Biomedical Optics Express, 11, 2705-2721(2020).

[22] Möckl L, Roy A R, Petrov P N et al. Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural network BGNet[J]. Proceedings of the National Academy of Sciences of the United States of America, 117, 60-67(2020).

[23] Kim T, Moon S, Xu K. Information-rich localization microscopy through machine learning[J]. Nature Communications, 10, 1996(2019).

[24] Jin L H, Liu B, Zhao F Q et al. Deep learning enables structured illumination microscopy with low light levels and enhanced speed[J]. Nature Communications, 11, 1934(2020).

[25] Shah Z H, Müller M, Wang T C et al. Deep-learning based denoising and reconstruction of super-resolution structured illumination microscopy images[J]. Photonics Research, 9, B168-B181(2021).

[26] Burns Z, Liu Z W. Untrained, physics-informed neural networks for structured illumination microscopy[J]. Optics Express, 31, 8714-8724(2023).

[27] Huang B Y, Li J, Yao B W et al. Enhancing image resolution of confocal fluorescence microscopy with deep learning[J]. PhotoniX, 4, 2(2023).

[28] Wang Y F, Kuang C F, Gu Z T et al. Time-gated stimulated emission depletion nanoscopy[J]. Optical Engineering, 52, 093107(2013).

[29] Hell S W, Wichmann J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy[J]. Optics Letters, 19, 780-782(1994).

[30] Heller I, Sitters G, Broekmans O D et al. STED nanoscopy combined with optical tweezers reveals protein dynamics on densely covered DNA[J]. Nature Methods, 10, 910-916(2013).

[31] Liu Y T, Zhou J J, Wen S H et al. On-chip mirror enhanced multiphoton upconversion super-resolution microscopy[J]. Nano Letters, 23, 5514-5519(2023).

[32] Takasaki K T, Ding J B, Sabatini B L. Live-cell superresolution imaging by pulsed STED two-photon excitation microscopy[J]. Biophysical Journal, 104, 770-777(2013).

[33] Kilian N, Goryaynov A, Lessard M D et al. Assessing photodamage in live-cell STED microscopy[J]. Nature Methods, 15, 755-756(2018).

[34] Tortarolo G, Castello M, Diaspro A et al. Evaluating image resolution in stimulated emission depletion microscopy[J]. Optica, 5, 32-35(2018).

[35] Chen Y I, Chang Y J, Sun Y S et al. Spatial resolution enhancement in photon-starved STED imaging using deep learning-based fluorescence lifetime analysis[J]. Nanoscale, 15, 9449-9456(2023).

[36] Pavani S R P, Thompson M A, Biteen J S et al. Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function[J]. Proceedings of the National Academy of Sciences of the United States of America, 106, 2995-2999(2009).

[37] Lew M D, Lee S F, Badieirostami M et al. Corkscrew point spread function for far-field three-dimensional nanoscale localization of pointlike objects[J]. Optics Letters, 36, 202-204(2011).

[38] Ji Y H, Chen D N, Wu H Z et al. Localizing axial dense emitters based on single-helix point spread function and deep learning[EB/OL]. http://arxiv.org/abs/2402.06863v2

[39] Pan W H, Li W, Qu J H et al. Research progress on organic fluorescent probes for single molecule localization microscopy[J]. Chinese Journal of Applied Chemistry, 36, 269-281(2019).

[40] Hess S T, Girirajan T P K, Mason M D. Ultra-high resolution imaging by fluorescence photoactivation localization microscopy[J]. Biophysical Journal, 91, 4258-4272(2006).

[41] Colabrese S, Castello M, Vicidomini G et al. Machine learning approach for single molecule localisation microscopy[J]. Biomedical Optics Express, 9, 1680-1691(2018).

[42] Sahel Y B, Eldar Y. Self-STORM: deep unrolled self-supervised learning for super-resolution microscopy[EB/OL]. http://arxiv.org/abs/2403.16974v1

[43] Li J Y, Tong G, Pan Y N et al. Spatial and temporal super-resolution for fluorescence microscopy by a recurrent neural network[J]. Optics Express, 29, 15747-15763(2021).

[44] Boyd N, Jonas E, Babcock H et al. DeepLoco: fast 3D localization microscopy using neural networks[EB/OL]. https://www.biorxiv.org/content/10.1101/267096v1

[45] Zelger P, Kaser K, Rossboth B et al. Three-dimensional localization microscopy using deep learning[J]. Optics Express, 26, 33166-33179(2018).

[46] Speiser A, Müller L R, Hoess P et al. Deep learning enables fast and dense single-molecule localization with high accuracy[J]. Nature Methods, 18, 1082-1090(2021).

[47] Ouyang W, Aristov A, Lelek M et al. Deep learning massively accelerates super-resolution localization microscopy[J]. Nature Biotechnology, 36, 460-468(2018).

[48] Isola P, Zhu J Y, Zhou T H et al. Image-to-image translation with conditional adversarial networks[C], 5967-5976(2017).

[49] Saguy A, Alalouf O, Opatovski N et al. DBlink: dynamic localization microscopy in super spatiotemporal resolution via deep learning[J]. Nature Methods, 20, 1939-1948(2023).

[50] Cascarano P, Comes M C, Sebastiani A et al. DeepCEL0 for 2D single-molecule localization in fluorescence microscopy[J]. Bioinformatics, 38, 1411-1419(2022).

[51] Nehme E, Freedman D, Gordon R et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning[J]. Nature Methods, 17, 734-740(2020).

[52] Zhang P Y, Liu S, Chaurasia A et al. Analyzing complex single-molecule emission patterns with deep learning[J]. Nature Methods, 15, 913-916(2018).

[53] Hershko E, Weiss L E, Michaeli T et al. Multicolor localization microscopy and point-spread-function engineering by deep learning[J]. Optics Express, 27, 6158-6183(2019).

[54] Jeong D, Kim D. Super-resolution fluorescence microscopy-based single-molecule spectroscopy[J]. Bulletin of the Korean Chemical Society, 43, 316-327(2022).

[55] Kim G H, Chung J, Park H et al. Single-molecule sensing by grating-based spectrally resolved super-resolution microscopy[J]. Bulletin of the Korean Chemical Society, 42, 270-278(2021).

[56] Zhang Z Y, Kenny S J, Hauser M et al. Ultrahigh-throughput single-molecule spectroscopy and spectrally resolved super-resolution microscopy[J]. Nature Methods, 12, 935-938(2015).

[57] Zhang Z Y, Zhang Y, Ying L et al. Machine-learning based spectral classification for spectroscopic single-molecule localization microscopy[J]. Optics Letters, 44, 5864-5867(2019).

[58] Brunstein M, Wicker K, Hérault K et al. Full-field dual-color 100-nm super-resolution imaging reveals organization and dynamics of mitochondrial and ER networks[J]. Optics Express, 21, 26162-26173(2013).

[59] Schermelleh L, Carlton P M, Haase S et al. Subdiffraction multicolor imaging of the nuclear periphery with 3D structured illumination microscopy[J]. Science, 320, 1332-1336(2008).

[60] Turcotte R, Liang Y J, Tanimoto M et al. Dynamic super-resolution structured illumination imaging in the living brain[J]. Proceedings of the National Academy of Sciences of the United States of America, 116, 9586-9591(2019).

[61] Ma Y, Wen K, Liu M et al. Recent advances in structured illumination microscopy[J]. Journal of Physics: Photonics, 3, 024009(2021).

[62] Saxena M, Eluru G, Gorthi S S. Structured illumination microscopy[J]. Advances in Optics and Photonics, 7, 241-275(2015).

[63] Ling C, Zhang C L, Wang M Q et al. Fast structured illumination microscopy via deep learning[J]. Photonics Research, 8, 1350-1359(2020).

[64] Qiao C, Chen X Y, Zhang S W et al. 3D structured illumination microscopy via channel attention generative adversarial network[J]. IEEE Journal of Selected Topics in Quantum Electronics, 27, 6801711(2021).

[65] Qiao C, Li D, Guo Y T et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy[J]. Nature Methods, 18, 194-202(2021).

[66] Cheng X, Li J, Dai Q et al. Fast and lightweight network for single frame structured illumination microscopy super-resolution[J]. IEEE Transactions on Instrumentation and Measurement, 71, 5007711(2022).

[67] Müller M, Mönkemöller V, Hennig S et al. Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ[J]. Nature Communications, 7, 10980(2016).

[68] Wicker K, Mandula O, Best G et al. Phase optimisation for structured illumination microscopy[J]. Optics Express, 21, 2032-2049(2013).

[69] Delp S L, Anderson F C, Arnold A S et al. OpenSim: open-source software to create and analyze dynamic simulations of movement[J]. IEEE Transactions on Bio-Medical Engineering, 54, 1940-1950(2007).

[70] Chen Y B, Liu Q Q, Zhang J F et al. Deep learning enables contrast-robust super-resolution reconstruction in structured illumination microscopy[J]. Optics Express, 32, 3316-3328(2024).

[71] Cao R Z, Chen Y H, Liu W J et al. Inverse matrix based phase estimation algorithm for structured illumination microscopy[J]. Biomedical Optics Express, 9, 5037-5051(2018).

[72] Liao J H, Zhang C S, Xu X C et al. Deep-MSIM: fast image reconstruction with deep learning in multifocal structured illumination microscopy[J]. Advanced Science, 10, e2300947(2023).

[73] Gustafsson M G L, Shao L, Carlton P M et al. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination[J]. Biophysical Journal, 94, 4957-4970(2008).

[74] Wang H D, Rivenson Y, Jin Y Y et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy[J]. Nature Methods, 16, 103-110(2019).

Tools

Get Citation

Copy Citation Text

Xinyi Lu, Yu Huang, Zitong Zhang, Tianxiao Wu, Hongjun Wu, Yongtao Liu, Zhong Fang, Chao Zuo, Qian Chen. Advances in Deep Learning for Super-Resolution Microscopy（Invited）[J]. Laser & Optoelectronics Progress, 2024, 61(16): 1611002

Download Citation

EndNote(RIS)BibTex Plain Text

Set citation alerts for article

Save article for my favorites

Paper Information

Category: Imaging Systems

Received: Jun. 7, 2024

Accepted: Jul. 2, 2024

Published Online: Aug. 19, 2024

The Author Email: Liu Yongtao (Yongtao.Liu@njust.edu.cn), Fang Zhong (fangzhong@njust.edu.cn), Zuo Chao (zuochao@njust.edu.cn)

DOI:10.3788/LOP241455

CSTR:32186.14.LOP241455

Topics

laser devices and laser physics

Lasers and Laser Optics

Laser physics

laser manufacturing

Instrumentation, Measurement and Metrology