Infrared and Laser Engineering, Volume. 51, Issue 11, 20220536(2022)

Performance enhancement of fluorescence microscopy by using deep learning (invited)

Zihan Xiong1,2, Liangfeng Song1,2, Xin Liu1, Chao Zuo1, and Peng Gao1
Author Affiliations
  • 1School of Physics, Xidian University, Xi’an 710071, China
  • 2Hangzhou Institute of Technology, Xidian University, Hangzhou 311200, China
  • show less
    References(99)

    [1] Hamilton N. Quantification and its applications in fluorescent microscopy imaging[J]. Traffic, 10, 951-961(2009).

    [2] Shi R, Jin C, Xie H, et al. Multi-plane, wide-field fluorescent microscopy for biodynamic imaging in vivo[J]. Biomed Opt Express, 10, 6625-6635(2019).

    [3] Goodman J W, Lawrence R. Digital image formation from electronically detected holograms[J]. Applied Physics Letters, 11, 77-79(1967).

    [4] Fan Y, Li J, Lu L, et al. Smart computational light microscopes (SCLMs) of smart computational imaging laboratory (SCILab)[J]. PhotoniX, 2, 1-64(2021).

    [5] Gao P, Yuan C. Resolution enhancement of digital holographic microscopy via synthetic aperture: a review[J]. Light: Advanced Manufacturing, 3, 105-120(2022).

    [6] Gao Peng, Wen Kai, Sun Xueying, et al. Review of resolution enhancement technologies in quantitative phase microscopy[J]. Infrared and Laser Engineering, 48, 0603007(2019).

    [7] Lichtman J W, Conchello J A. Fluorescence microscopy[J]. Nature Methods, 2, 910-919(2005).

    [8] Conchello J A, Lichtman J W. Optical sectioning microscopy[J]. Nature Methods, 2, 920-931(2005).

    [9] Murfin K E, Chaston J, Goodrich-Blair H. Visualizing bacteria in nematodes using fluorescent microscopy[J]. Journal of Visualized Experiments, 68, e4298(2012).

    [10] Mickoleit M, Schmid B, Weber M, et al. High-resolution reconstruction of the beating zebrafish heart[J]. Nature Methods, 11, 919-922(2014).

    [11] Giepmans B N, Adams S R, Ellisman M H, et al. The fluorescent toolbox for assessing protein location and function[J]. Science, 312, 217-224(2006).

    [12] Palmer A E, Tsien R Y. Measuring calcium signaling using genetically targetable fluorescent indicators[J]. Nature Protocols, 1, 1057-1065(2006).

    [13] Boulanger J, Kervrann C, Bouthemy P, et al. Patch-based nonlocal functional for denoising fluorescence microscopy image sequences[J]. IEEE Transactions on Medical Imaging, 29, 442-454(2009).

    [14] Betzig E, Patterson G H, Sougrat R, et al. Imaging intracellular fluorescent proteins at nanometer resolution[J]. Science, 313, 1642-1645(2006).

    [15] Hell S W, Wichmann J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluo-rescence microscopy[J]. Optics Letters, 19, 780-782(1994).

    [16] Gustafsson M G. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy[J]. Journal of Microscopy, 198, 82-87(2000).

    [17] Gao P, Prunsche B, Zhou L, et al. Background suppression in fluorescence nanoscopy with stimulated emission double depletion[J]. Nature Photonics, 11, 163-169(2017).

    [18] Klar T A, Jakobs S, Dyba M, et al. Fluorescence microscopy with diffraction resolution barrier broken by stimulated emission[J]. Proc Natl Acad Sci USA, 97, 8206-8210(2000).

    [19] Shroff H, Galbraith C G, Galbraith J A, et al. Live-cell photoactivated localization microscopy of nanoscale adhesion dynamics[J]. Nat Methods, 5, 417-423(2008).

    [20] Mertz J. Optical sectioning microscopy with planar or structured illumination[J]. Nature Methods, 8, 811-819(2011).

    [21] Icha J, Weber M, Waters J C, et al. Phototoxicity in live fluorescence microscopy, and how to avoid it[J]. Bioessays, 39, 1700003(2017).

    [22] Helmerich D A, Beliu G, Matikonda S S, et al. Photoblueing of organic dyes can cause artifacts in super-resolution microscopy[J]. Nature Methods, 18, 253-257(2021).

    [23] [23] Wang SC. Artificial Neural wk [M]Interdisciplinary Computing in Java Programming. Boston, MA: Springer, 2003: 81100.

    [24] Shin H C, Roth H R, Gao M, et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning[J]. IEEE Trans Med Imaging, 35, 1285-1298(2016).

    [25] Wang H, Rivenson Y, Jin Y, et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy[J]. Nature Methods, 16, 103-110(2019).

    [26] Zhang H, Fang C, Xie X, et al. High-throughput, high-resolution deep learning microscopy based on registration-free generative adversarial network[J]. Biomed Opt Express, 10, 1044-1063(2019).

    [27] Zhou H, Cai R, Quan T, et al. 3D high resolution generative deep-learning network for fluorescence microscopy imaging[J]. Optics Letters, 45, 1695-1698(2020).

    [28] Li M Z, Shan H M, Pryshchep S, et al. Deep adversarial network for super stimulated emission depletion imaging[J]. Journal of Nanophotonics, 14, 016009(2020).

    [29] Christensen C N, Ward E N, Lio P, et al. ML-SIM: Universal reconstruction of structured illumination microscopy images using transfer learning[J]. Biomedical Optics Express, 12, 2720-2733(2021).

    [30] Shah Z H, Müller M, Wang T C, et al. Deep-learning based denoising and reconstruction of super-resolution structured illumination microscopy images[J]. Photonics Research, 9, B168-B181(2021).

    [31] Jin L, Liu B, Zhao F, et al. Deep learning enables structured illumination microscopy with low light levels and enhanced speed[J]. Nature Communications, 11, 1934(2020).

    [32] Ling C, Zhang C L, Wang M Q, et al. Fast structured illumination microscopy via deep learning[J]. Photonics Research, 8, 1350-1359(2020).

    [33] [33] Boyd N, Jonas E, Babcock H, et al. DeepLoco: Fast 3D localization microscopy using neural wks [ZOL]. bixiv, (20180226)[20220801]. https:doi.g10.1101267096.

    [34] Nehme E, Weiss L E, Michaeli T, et al. Deep-STORM: super-resolution single-molecule microscopy by deep learning[J]. Optica, 5, 458-464(2018).

    [35] Speiser A, Müller L R, Hoess P, et al. Deep learning enables fast and dense single-molecule localization with high accuracy[J]. Nature Methods, 18, 1082-1090(2021).

    [36] Weigert M, Schmidt U, Boothe T, et al. Content-aware image restoration: pushing the limits of fluorescence microscopy[J]. Nature Methods, 15, 1090-1097(2018).

    [37] Wang Z, Zhu L, Zhang H, et al. Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning[J]. Nature Methods, 18, 551-556(2021).

    [38] Zhang X, Chen Y, Ning K, et al. Deep learning optical-sectioning method[J]. Optics Express, 26, 30762-30772(2018).

    [39] Bai C, Liu C, Yu X H, et al. Imaging enhancement of light-sheet fluorescence microscopy via deep learning[J]. IEEE Photonics Technology Letters, 31, 1803-1806(2019).

    [40] Huang L, Chen H, Luo Y, et al. Recurrent neural network-based volumetric fluorescence microscopy[J]. Light Sci Appl, 10, 62(2021).

    [41] Wu Y C, Rivenson Y, Wang H D, et al. Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning[J]. Nature Methods, 16, 1323-1331(2019).

    [42] Ning K, Zhang X, Gao X, et al. Deep-learning-based whole-brain imaging at single-neuron resolution[J]. Biomedical Optics Express, 11, 3567-3584(2020).

    [43] Bai C, Yu X, Peng T, et al. 3D imaging restoration of spinning-disk confocal microscopy via deep learning[J]. IEEE Photonics Technology Letters, 32, 1131-1134(2020).

    [44] Zhang H, Zhao Y, Fang C, et al. Exceeding the limits of 3D fluorescence microscopy using a dual-stage-processing network[J]. Optica, 7, 1627-1640(2020).

    [45] Hu L, Hu S, Gong W, et al. Image enhancement for fluorescence microscopy based on deep learning with prior knowledge of aberration[J]. Optics Letters, 46, 2055-2058(2021).

    [46] Xiao L, Fang C, Zhu L, et al. Deep learning-enabled efficient image restoration for 3D microscopy of turbid biological specimens[J]. Optics Express, 28, 30234-30247(2020).

    [47] Lecun Y, Bengio Y, Hinton G. Deep learning[J]. Nature, 521, 436-444(2015).

    [48] Mcculloch W S, Pitts W. A logical calculus of the ideas immanent in nervous activity[J]. The Bulletin of Mathematical Biophysics, 5, 115-133(1943).

    [49] [49] Lecun Y. A theetical framewk f backpropagation[C]Proceedings of the 1988 Connectionist Models Summer School, 1988: 2128.

    [50] Hinton G E, Salakhutdinov R R. Reducing the dimensionality of data with neural networks[J]. Science, 313, 504-507(2006).

    [51] [51] Krizhevsky A, Sutskever I, Hinton G. Image classification with deep convolutional neural wks[C]NIPS''12: Proceedings of the 25th International Conference on Neural Infmation Processing Systems, 2012, 1: 1097–1105.

    [52] Ongie G, Jalal A, Metzler C A, et al. Deep learning techniques for inverse problems in imaging[J]. IEEE Journal on Selected Areas in Information Theory, 1, 39-56(2020).

    [53] Ghosh N, Bhattacharya K. Cube beam-splitter interferometer for phase shifting interferometry[J]. Journal of Optics, 38, 191-198(2009).

    [54] Mccann M T, Jin K H, Unser M. Convolutional neural networks for inverse problems in imaging: A review[J]. IEEE Signal Processing Magazine, 34, 85-95(2017).

    [55] [55] O''shea K, Nash R. An introduction to convolutional neural wks [EBOL]. (20151126)[20220801]. https:arxiv.gabs1511.08458.

    [56] [56] Pang S, Du A, gun M A, et al. Beyond CNNs: exploiting further inherent symmetries in medical images f segmentation [EBOL]. (20200508)[20220801]. https:arxiv.gabs2005.03924.

    [57] [57] Ronneberger O, Fischer P, Brox T. U: Convolutional wks f biomedical image segmentation[C]International Conference on Medical Image Computing Computerassisted Intervention, 2015: 234241.

    [58] [58] Feizabadi M M, Shujjat A M, Shahid S, et al. Interactive latent interpolation on MNIST dataset [EBOL]. (20201015)[20220801]. https:arxiv.gabs2010.07581.

    [59] Zhu Linlin, Han Lu, Du Hong, et al. Multi-active contour cell segmentation method based on U-Net network[J]. Infrared and Laser Engineering, 49, 20200121(2020).

    [60] Medsker L R, Jain L. Recurrent neural networks[J]. Design and Applications, 5, 64-67(2001).

    [61] Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural Computation, 9, 1735-1780(1997).

    [62] [62] Vinyals O, Toshev A, Bengio S, et al. Show tell: A neural image caption generat[C]Proceedings of the IEEE conference on computer vision pattern recognition, 2015: 31563164.

    [63] [63] Sutskever I, Vinyals O, Le Q V. Sequence to sequence learning with neural wks [EBOL]. (20140910)[20220801]. https:arxiv.gabs1409.3215.

    [64] [64] Graves A. Generating sequences with recurrent neural wks [EBOL]. (20130804)[20220801]. https:arxiv.gabs1308.0850v5.

    [65] Sajjad M, Kwon S. Clustering-based speech emotion recognition by incorporating learned features and deep BiLSTM[J]. IEEE Access, 8, 79861-79875(2020).

    [66] [66] Goodfellow I, PougetAbadie J, Mirza M, et al. Generative adversarial s [EBOL]. (20140610)[20220801]. https:arxiv.gabs1406.2661.

    [67] [67] Isola P, Zhu JY, Zhou T, et al. Imagetoimage translation with conditional adversarial wks[C]Proceedings of the IEEE Conference On Computer Vision Pattern Recognition, 2017: 11251134.

    [68] [68] Brock A, Donahue J, Simonyan K. Large scale GAN training f high fidelity natural image synthesis [EBOL]. (20180928)[20220801]. https:arxiv.gabs1809.11096v2.

    [69] [69] Cao J, Hou L, Yang MH, et al. Remix: Towards imagetoimage translation with limited data[C]Proceedings of the IEEECVF Conference on Computer Vision Pattern Recognition, 2021: 1501815027.

    [70] [70] Wang X, Yu K, Wu S, et al. Esrgan: Enhanced superresolution generative adversarial wks[C]Proceedings of the European Conference On Computer Vision (ECCV) Wkshops, 2018.

    [71] Abbe E. Beiträge zur theorie des mikroskops und der mikroskopischen wahrnehmung[J]. Archiv für Mikroskopische Anatomie, 9, 413-418(1873).

    [72] [72] Pawley J. Hbook of Biological Confocal Microscopy [M]. New Yk: Springer Science & Business Media, 2006.

    [73] Ji Wei, Xu Tao, Liu Bei. Super-resolution fluorescent micro-scopy: A brief introduction to the Nobel Prize in Chemistry 2014[J]. Chinese Journal of Nature, 36, 404-408(2014).

    [74] Rust M J, Bates M, Zhuang X J N M. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM)[J]. Nature Methods, 3, 793-796(2006).

    [75] Heintzmann R, Huser T. Super-resolution structured illumi-nation microscopy[J]. Chemical Reviews, 117, 13890-13908(2017).

    [76] Tam J, Merino D. Stochastic optical reconstruction microscopy (STORM) in comparison with stimulated emission depletion (STED) and other imaging methods[J]. J Neurochem, 135, 643-658(2015).

    [77] Huang B, Bates M, Zhuang X. Super-resolution fluorescence microscopy[J]. Annu Rev Biochem, 78, 993-1016(2009).

    [78] Schermelleh L, Heintzmann R, Leonhardt H. A guide to super-resolution fluorescence microscopy[J]. J Cell Biol, 190, 165-175(2010).

    [79] Nguyen J P, Shipley F B, Linder A N, et al. Whole-brain calcium imaging with cellular resolution in freely behaving Caenorhabditis elegans[J]. Proc Natl Acad Sci USA, 113, E1074-1081(2016).

    [80] Juette M F, Gould T J, Lessard M D, et al. Three-dimensional sub-100 nm resolution fluorescence microscopy of thick samples[J]. Nature Methods, 5, 527-529(2008).

    [81] Prabhat P, Ram S, Ward E S, et al. Simultaneous imaging of different focal planes in fluorescence microscopy for the study of cellular dynamics in three dimensions[J]. IEEE Transactions on NanoBioscience, 3, 237-242(2004).

    [82] Johnson C, Exell J, Kuo J, et al. Continuous focal translation enhances rate of point-scan volumetric microscopy[J]. Optics Express, 27, 36241-36258(2019).

    [83] Li H, Guo C, Kim-Holzapfel D, et al. Fast, volumetric live-cell imaging using high-resolution light-field microscopy[J]. Biomedical Optics Express, 10, 29-49(2019).

    [84] Pascucci M, Ganesan S, Tripathi A, et al. Compressive three-dimensional super-resolution microscopy with speckle-saturated fluorescence excitation[J]. Nature Communications, 10, 1327(2019).

    [85] Gong H, Xu D, Yuan J, et al. High-throughput dual-colour precision imaging for brain-wide connectome with cytoarchitectonic landmarks at the cellular level[J]. Nature Communications, 7, 1-12(2016).

    [86] Carlton P M, Boulanger J, Kervrann C, et al. Fast live simultaneous multiwavelength four-dimensional optical microscopy[J]. Proceedings of the National Academy of Sciences, 107, 16016-16022(2010).

    [87] Luisier F, Blu T, Unser M. Image denoising in mixed Poisson-Gaussian noise[J]. IEEE Trans Image Process, 20, 696-708(2011).

    [88] Soubies E, Soulez F, Mccann M T, et al. Pocket guide to solve inverse problems with GlobalBioIm[J]. Inverse Problems, 35, 104006(2019).

    [89] Arigovindan M, Fung J C, Elnatan D, et al. High-resolution restoration of 3D structures from widefield images with extreme low signal-to-noise-ratio[J]. Proceedings of the National Academy of Sciences, 110, 17344-17349(2013).

    [90] Setzer S, Steidl G, Teuber T. Deblurring Poissonian images by split Bregman techniques[J]. Journal of Visual Commu-nication and Image Representation, 21, 193-199(2010).

    [91] [91] Zhang Y, Zhu Y, Nichols E, et al. A poissongaussian denoising dataset with real fluescence microscopy images[C]Proceedings of the IEEECVF Conference on Computer Vision Pattern Recognition, 2019: 1171011718.

    [92] Hagen G M, Bendesky J, Machado R, et al. Fluorescence microscopy datasets for training deep neural networks[J]. GigaScience, 10, giab032(2021).

    [93] Belthangady C, Royer L A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction[J]. Nat Methods, 16, 1215-1225(2019).

    [94] Christiansen E M, Yang S J, Ando D M, et al. In silico labeling: predicting fluorescent labels in unlabeled images[J]. Cell, 173, 792-803e719(2018).

    [95] Ounkomol C, Seshamani S, Maleckar M M, et al. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy[J]. Nature Methods, 15, 917-920(2018).

    [96] Khater I M, Aroca-Ouellette S T, Meng F, et al. Caveolae and scaffold detection from single molecule localization microscopy data using deep learning[J]. PLoS One, 14, e0211659(2019).

    [97] Li X, Zhang G, Qiao H, et al. Unsupervised content-preserving transformation for optical microscopy[J]. Light Sci Appl, 10, 44(2021).

    [98] [98] Chen X, Kel M E, He S, et al. Artificial confocal microscopy f deep labelfree imaging [EBOL]. (20211028)[20220801]. https:arxiv.gabs2110.14823.

    [99] [99] Robitaille L É, Dur A, Gardner MA, et al. Learning to become an expert: Deep wks applied to superresolution microscopy[C]ThirtySecond AAAI Conference on Artificial Intelligence, 2018.

    Tools

    Get Citation

    Copy Citation Text

    Zihan Xiong, Liangfeng Song, Xin Liu, Chao Zuo, Peng Gao. Performance enhancement of fluorescence microscopy by using deep learning (invited)[J]. Infrared and Laser Engineering, 2022, 51(11): 20220536

    Download Citation

    EndNote(RIS)BibTexPlain Text
    Save article for my favorites
    Paper Information

    Category: Special issue-Fluorescence microscopy: techniques and applications

    Received: Aug. 1, 2022

    Accepted: --

    Published Online: Feb. 9, 2023

    The Author Email:

    DOI:10.3788/IRLA20220536

    Topics