Laser & Optoelectronics Progress, Volume. 58, Issue 18, 1811007(2021)

Deep Learning Based Fluorescence Microscopy Imaging Technologies and Applications

Haoyu Li, Liying Qu, Zijie Hua, Xinwei Wang, Weisong Zhao, and Jian Liu*
Author Affiliations
  • Advanced Microscopy and Instrumentation Research Center, School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin, Heilongjiang 150080, China
  • show less
    References(141)

    [1] Lichtman J W, Conchello J A. Fluorescence microscopy[J]. Nature Methods, 2, 910-919(2005).

    [2] Huisken J, Swoger J, del Bene F et al. Optical sectioning deep inside live embryos by selective plane illumination microscopy[J]. Science, 305, 1007-1009(2004).

    [3] Chen B C, Legant W R, Wang K et al. Lattice light-sheet microscopy: imaging molecules to embryos at high spatiotemporal resolution[J]. Science, 346, 1257998(2014).

    [5] Gustafsson M G. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy[J]. Journal of Microscopy, 198, 82-87(2000).

    [6] Betzig E, Patterson G H, Sougrat R et al. Imaging intracellular fluorescent proteins at nanometer resolution[J]. Science, 313, 1642-1645(2006).

    [7] Rust M J, Bates M, Zhuang X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM)[J]. Nature Methods, 3, 793-795(2006).

    [8] Hell S W, Wichmann J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy[J]. Optics Letters, 19, 780-782(1994).

    [9] Sigal Y M, Zhou R B, Zhuang X W. Visualizing and discovering cellular structures with super-resolution microscopy[J]. Science, 361, 880-887(2018).

    [10] Nixon-Abell J, Obara C J, Weigel A V et al. Increased spatiotemporal resolution reveals highly dynamic dense tubular matrices in the peripheral ER[J]. Science, 354, aaf3928(2016).

    [11] Valm A M, Cohen S, Legant W R et al. Applying systems-level spectral imaging and analysis to reveal the organelle interactome[J]. Nature, 546, 162-167(2017).

    [12] Hess S T, Gould T J, Gudheti M V et al. Dynamic clustered distribution of hemagglutinin resolved at 40 nm in living cell membranes discriminates between raft theories[J]. Proceedings of the National Academy of Sciences of USA, 104, 17370-17375(2007).

    [13] Shroff H, Galbraith C G, Galbraith J A et al. Live-cell photoactivated localization microscopy of nanoscale adhesion dynamics[J]. Nature Methods, 5, 417-423(2008).

    [14] Westphal V, Rizzoli S O, Lauterbach M A et al. Video-rate far-field optical nanoscopy dissects synaptic vesicle movement[J]. Science, 320, 246-249(2008).

    [15] Shin W, Ge L H, Arpino G et al. Visualization of membrane pore in live cells reveals a dynamic-pore theory governing fusion and endocytosis[J]. Cell, 173, 934-945(2018).

    [17] Huang X, Fan J, Li L et al. Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy[J]. Nature Biotechnology, 36, 451-459(2018).

    [18] Li D, Shao L, Chen B C et al. Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics[J]. Science, 349, aab3500(2015).

    [19] Guo Y T, Li D, Zhang S W et al. Visualizing intracellular organelle and cytoskeletal interactions at nanoscale resolution on millisecond timescales[J]. Cell, 175, 1430-1442(2018).

    [20] Jacquemet G, Carisey A F, Hamidi H et al. The cell biologist’s guide to super-resolution microscopy[J]. Journal of Cell Science, 133, jcs.240713(2020).

    [21] Icha J, Weber M, Waters J C et al. Phototoxicity in live fluorescence microscopy, and how to avoid it[J]. BioEssays, 39, 1700003(2017).

    [22] Laissue P P, Alghamdi R A, Tomancak P et al. Assessing phototoxicity in live fluorescence imaging[J]. Nature Methods, 14, 657-661(2017).

    [23] Mortensen K I, Churchman L S, Spudich J A et al. Optimized localization analysis for single-molecule tracking and super-resolution microscopy[J]. Nature Methods, 7, 377-381(2010).

    [24] Dempsey G T, Vaughan J C, Chen K H et al. Evaluation of fluorophores for optimal performance in localization-based super-resolution imaging[J]. Nature Methods, 8, 1027-1036(2011).

    [25] Helmerich D A, Beliu G, Matikonda S S et al. Photoblueing of organic dyes can cause artifacts in super-resolution microscopy[J]. Nature Methods, 18, 253-257(2021).

    [26] Erdélyi M, Sinkó J, Kákonyi R et al. Origin and compensation of imaging artefacts in localization-based super-resolution microscopy[J]. Methods, 88, 122-132(2015).

    [27] Schaefer L H, Schuster D, Schaffer J. Structured illumination microscopy: artefact analysis and reduction utilizing a parameter optimization approach[J]. Journal of Microscopy, 216, 165-174(2004).

    [28] Demmerle J, Innocent C, North A J et al. Strategic and practical guidelines for successful structured illumination microscopy[J]. Nature Protocols, 12, 988-1010(2017).

    [29] Mo Y, Feng F, Mao H et al. Structured illumination microscopy artefacts caused by illumination scattering[J]. Philosophical Transactions of the Royal Society A, 379, 20200153(2021).

    [30] Gwosch K C, Pape J K, Balzarotti F et al. MINFLUX nanoscopy delivers 3D multicolor nanometer resolution in cells[J]. Nature Methods, 17, 217-224(2020).

    [31] Bodén A, Pennacchietti F, Coceano G et al. Volumetric live cell imaging with three-dimensional parallelized RESOLFT microscopy[J]. Nature Biotechnology, 39, 609-618(2021).

    [32] Hao X, Allgeyer E S, Lee D R et al. Three-dimensional adaptive optical nanoscopy for thick specimen imaging at sub-50-nm resolution[J]. Nature Methods, 18, 688-693(2021).

    [33] Liu T L, Upadhyayula S, Milkie D E et al. Observing the cell in its native state: imaging subcellular dynamics in multicellular organisms[J]. Science, 360, 1-15(2018).

    [34] Lin R Z, Kipreos E T, Zhu J et al. Subcellular three-dimensional imaging deep through multicellular thick samples by structured illumination microscopy and adaptive optics[J]. Nature Communications, 12, 3148(2021).

    [35] Mahecic D, Gambarotto D, Douglass K M et al. Homogeneous multifocal excitation for high-throughput super-resolution imaging[J]. Nature Methods, 17, 726-733(2020).

    [36] Douglass K M, Sieben C, Archetti A et al. Super-resolution imaging of multiple cells by optimized flat-field epi-illumination[J]. Nature Photonics, 10, 705-708(2016).

    [37] Yu J Y, Becker S R, Folberth J et al. Achieving superresolution with illumination-enhanced sparsity[J]. Optics Express, 26, 9850-9865(2018).

    [38] Tenne R, Rossman U, Rephael B et al. Super-resolution enhancement by quantum image scanning microscopy[J]. Nature Photonics, 13, 116-122(2019).

    [39] Masullo L A, Bodén A, Pennacchietti F et al. Enhanced photon collection enables four dimensional fluorescence nanoscopy of living systems[J]. Nature Communications, 9, 3281(2018).

    [40] Dreier J, Castello M, Coceano G et al. Smart scanning for low-illumination and fast RESOLFT nanoscopy in vivo[J]. Nature Communications, 10, 556(2019).

    [41] Zhang Y, Schroeder L K, Lessard M D et al. Nanoscale subcellular architecture revealed by multicolor three-dimensional salvaged fluorescence imaging[J]. Nature Methods, 17, 225-231(2020).

    [42] Zhang Z, Kenny S J, Hauser M et al. Ultrahigh-throughput single-molecule spectroscopy and spectrally resolved super-resolution microscopy[J]. Nature Methods, 12, 935-938(2015).

    [43] Belthangady C, Royer L A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction[J]. Nature Methods, 16, 1215-1225(2019).

    [44] Gustafsson N, Culley S, Ashdown G et al. Fast live-cell conventional fluorophore nanoscopy with ImageJ through super-resolution radial fluctuations[J]. Nature Communications, 7, 12471(2016).

    [45] Dertinger T, Colyer R, Iyer G et al. Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI)[J]. Proceedings of the National Academy of Sciences of USA, 106, 22287-22292(2009).

    [46] Agarwal K, Macháň R. Multiple signal classification algorithm for super-resolution fluorescence microscopy[J]. Nature Communications, 7, 13752(2016).

    [47] Cox S, Rosten E, Monypenny J et al. Bayesian localization microscopy reveals nanoscale podosome dynamics[J]. Nature Methods, 9, 195-200(2011).

    [49] Yahiatene I, Hennig S, Müller M et al. Entropy-based super-resolution imaging (ESI): from disorder to fine detail[J]. ACS Photonics, 2, 1049-1056(2015).

    [50] Richardson W H. Bayesian-based iterative method of image restoration[J]. Journal of the Optical Society of America, 62, 55-59(1972).

    [51] Arigovindan M, Fung J C, Elnatan D et al. High-resolution restoration of 3D structures from widefield images with extreme low signal-to-noise-ratio[J]. Proceedings of the National Academy of Sciences of the USA, 110, 17344-17349(2013).

    [54] Aspelmeier T, Egner A, Munk A. Modern statistical challenges in high-resolution fluorescence microscopy[J]. Annual Review of Statistics and Its Application, 2, 163-202(2015).

    [55] Blasse C, Saalfeld S, Etournay R et al. PreMosa: extracting 2D surfaces from 3D microscopy mosaics[J]. Bioinformatics, 33, 2563-2569(2017).

    [56] Shihavuddin A, Basu S, Rexhepaj E et al. Smooth 2D manifold extraction from 3D image stack[J]. Nature Communications, 8, 15554(2017).

    [57] LeCun Y, Bengio Y, Hinton G. Deep learning[J]. Nature, 521, 436-444(2015).

    [58] Waller L, Tian L. Machine learning for 3D microscopy[J]. Nature, 523, 416-417(2015).

    [59] Cortes C, Vapnik V. Support-vector networks[J]. Machine Learning, 20, 273-297(1995).

    [60] Reynolds D A, Quatieri T F, Dunn R B. Speaker verification using adapted Gaussian mixture models[J]. Digital Signal Processing, 10, 19-41(2000).

    [61] Alam F I, Zhou J, Liew A W C et al. Conditional random field and deep feature learning for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 57, 1612-1628(2019).

    [62] Rabiner L R. A tutorial on hidden Markov models and selected applications in speech recognition[J]. Proceedings of the IEEE, 77, 257-286(1989).

    [63] Hosmer D W, Jr, Lemeshow S, Sturdivant R X. Applied logistic regression[M](2013).

    [64] McCulloch W S, Pitts W. A logical calculus of the ideas immanent in nervous activity[J]. The Bulletin of Mathematical Biophysics, 5, 115-133(1943).

    [65] Turing A M. Computing machinery and intelligence[M]. //Epstein R, Roberts G, Beber G. Parsing the Turing test, 23-65(2007).

    [66] Rosenblatt F. The perceptron: a probabilistic model for information storage and organization in the brain[J]. Psychological Review, 65, 386-408(1958).

    [67] Widrow B. Adaptive sampled-data systems[J]. IFAC Proceedings Volumes, 1, 433-439(1960).

    [68] Minsky M, Papert S A. Perceptrons: an introduction to computational geometry[M](2017).

    [69] Werbos P. Beyond regression: new tools for prediction and analysis in the behavioral sciences[D](1974).

    [70] Fukushima K, Miyake S. Neocognitron: a self-organizing neural network model for a mechanism of visual pattern recognition[M]. //Amari S I, Arbib M A. Competition and cooperation in neural nets. Lecture notes in computer science, 45, 267-285(1982).

    [71] Kohonen T. Self-organized formation of topologically correct feature maps[J]. Biological Cybernetics, 43, 59-69(1982).

    [72] Hopfield J J. Neural networks and physical systems with emergent collective computational abilities[J]. Proceedings of the National Academy of Sciences of the USA, 79, 2554-2558(1982).

    [73] Hinton G E, Sejnowski T J. Learning and relearning in Boltzmann machines[J]. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, 1, 282-317(1986).

    [74] Rumelhart D E, McClelland J L. PDP Research Group. Parallel distributed processing[M](1986).

    [75] Jordan M I. Serial order: a parallel distributed processing approach[J]. Advances in Psychology, 121, 471-495(1997).

    [76] Rumelhart D E, Hinton G E, Williams R J. Learning internal representations by error propagation[M]. //Readings in cognitive science: a perspective from psychology and artificial intelligence, 399-421(1988).

    [77] LeCun Y, Bottou L, Bengio Y et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 86, 2278-2324(1998).

    [78] Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural Computation, 9, 1735-1780(1997).

    [79] Schuster M, Paliwal K K. Bidirectional recurrent neural networks[J]. IEEE Transactions on Signal Processing, 45, 2673-2681(1997).

    [80] Hinton G E, Osindero S, Teh Y W. A fast learning algorithm for deep belief nets[J]. Neural Computation, 18, 1527-1554(2006).

    [81] Salakhutdinov R, Hinton G E. Deep Boltzmann machines[J]. Journal of Machine Learning Research, 5, 448-455(2009).

    [82] Nair V, Hinton G E. Rectified linear units improve restricted Boltzmann machines[C]. //Proceedings of the 27th Annual International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel. [S.l.:s.n.], 807-814(2010).

    [85] Silver D, Huang A, Maddison C J et al. Mastering the game of Go with deep neural networks and tree search[J]. Nature, 529, 484-489(2016).

    [87] Bengio Y. Deep learning of representations: looking forward[C]. //International Conference on Statistical Language and Speech Processing (SLSP), July 29-31, 2013, Tarragona, Spain, 1-37(2013).

    [88] Deng L, Yu D. Deep learning: methods and applications[J]. Foundations and Trends ® in Signal Processing, 7, 197-387(2014).

    [89] Szepesvári C. Algorithms for reinforcement learning[J]. Synthesis Lectures on Artificial Intelligence and Machine Learning, 4, 1-103(2010).

    [90] Bai C, Liu C, Yu X H et al. Imaging enhancement of light-sheet fluorescence microscopy via deep learning[J]. IEEE Photonics Technology Letters, 31, 1803-1806(2019).

    [91] Zhang H, Zhao Y X, Fang C Y et al. Exceeding the limits of 3D fluorescence microscopy using a dual-stage-processing network[J]. Optica, 7, 1627-1640(2020).

    [92] Wagner N, Beuttenmueller F, Norlin N et al. Deep learning-enhanced light-field imaging with continuous validation[J]. Nature Methods, 18, 557-563(2021).

    [93] Ling C, Zhang C L, Wang M Q et al. Fast structured illumination microscopy via deep learning[J]. Photonics Research, 8, 1350-1359(2020).

    [95] Nehme E, Weiss L E, Michaeli T et al. Deep-STORM: super-resolution single-molecule microscopy by deep learning[J]. Optica, 5, 458-464(2018).

    [96] Nehme E, Freedman D, Gordon R et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning[J]. Nature Methods, 17, 734-740(2020).

    [97] Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation[M]. //Navab N, Hornegger J, Wells W M, et al.Medical image computing and computer-assisted intervention-MICCAI 2015. Lecture notes in computer science, 9351, 234-241(2015).

    [98] Weigert M, Schmidt U, Boothe T et al. Content-aware image restoration: pushing the limits of fluorescence microscopy[J]. Nature Methods, 15, 1090-1097(2018).

    [99] Fang L, Monroe F, Novak S W et al. Deep learning-based point-scanning super-resolution imaging[J]. Nature Methods, 18, 406-416(2021).

    [100] Zhao F, Zhu L X, Fang C Y et al. Deep-learning super-resolution light-sheet add-on microscopy (Deep-SLAM) for easy isotropic volumetric imaging of large biological specimens[J]. Biomedical Optics Express, 11, 7273-7285(2020).

    [101] Wang Z, Zhu L, Zhang H et al. Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning[J]. Nature Methods, 18, 551-556(2021).

    [102] Jin L, Liu B, Zhao F et al. Deep learning enables structured illumination microscopy with low light levels and enhanced speed[J]. Nature Communications, 11, 1934(2020).

    [103] Ouyang W, Aristov A, Lelek M et al. Deep learning massively accelerates super-resolution localization microscopy[J]. Nature Biotechnology, 36, 460-468(2018).

    [106] Li X, Zhang G, Qiao H et al. Unsupervised content-preserving transformation for optical microscopy[J]. Light, Science & Applications, 10, 44(2021).

    [107] Li X, Dong J Y, Li B W et al. Fast confocal microscopy imaging based on deep learning[C]. //2020 IEEE International Conference on Computational Photography (ICCP), April 24-26, 2020, St. Louis, MO, USA., 1-12(2020).

    [108] Wang H D, Rivenson Y, Jin Y Y et al. Deep-learning enables cross-modality super-resolution in fluorescence microscopy[J]. Nature Methods, 16, 103-110(2019).

    [109] Ling C, Du L P, Yuan X C. Deep learning approach for nonlinear structured illumination microscopy[J]. Proceedings of SPIE, 11565, 115650P(2020).

    [110] Qiao C, Li D, Guo Y et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy[J]. Nature Methods, 18, 194-202(2021).

    [111] Li M Z, Shan H M, Pryshchep S et al. Deep adversarial network for super stimulated emission depletion imaging[J]. Journal of Nanophotonics, 14, 016009(2020).

    [112] Zhang Y L, Li K P, Li K et al. Image super-resolution using very deep residual channel attention networks[M]. //Ferrari V, Hebert M, Sminchisescu C, et al. Computer vision-ECCV 2018. Lecture notes in computer science, 11211, 294-310(2018).

    [113] Vaswani A, Shazeer N, Parmar N et al. Attention is all you need[C]. //Advances in Neural Information Processing Systems(NIPS), December 4-9, 2017, Long Beach, Canada, 5998-6008(2017).

    [114] Christensen C N, Ward E N, Lu M et al. ML-SIM: universal reconstruction of structured illumination microscopy images using transfer learning[J]. Biomedical Optics Express, 12, 2720-2733(2021).

    [115] Rivenson Y, Göröcs Z, Günaydin H et al. Deep learning microscopy[J]. Optica, 4, 1437-1443(2017).

    [116] Jonkman J, Brown C M, Wright G D et al. Tutorial: guidance for quantitative confocal microscopy[J]. Nature Protocols, 15, 1585-1611(2020).

    [117] Santi P A. Light sheet fluorescence microscopy: a review[J]. The Journal of Histochemistry and Cytochemistry, 59, 129-138(2011).

    [118] Yu X H, Liu C, Bai C et al. Progress in light-sheet fluorescence microscopy and applications[J]. Laser & Optoelectronics Progress, 57, 100001(2020).

    [119] Levoy M, Ng R, Adams A et al. Light field microscopy[C]. //ACM SIGGRAPH 2006 Papers on-SIGGRAPH’06, July 30-August 3, 2006. Boston, Massachusetts., 924-934(2006).

    [121] Saxena M, Eluru G, Gorthi S S. Structured illumination microscopy[J]. Advances in Optics and Photonics, 7, 241-275(2015).

    [122] Schermelleh L, Ferrand A, Huser T et al. Super-resolution microscopy demystified[J]. Nature Cell Biology, 21, 72-84(2019).

    [123] Wang Y, Zhao W, Wang K et al. Fluorescent nanoparticles for stimulated emission depletion microscopy[J]. Journal of Nanoscience and Nanotechnology, 20, 2308-2315(2020).

    [125] Manley S, Gillette J M, Patterson G H et al. High-density mapping of single-molecule trajectories with photoactivated localization microscopy[J]. Nature Methods, 5, 155-157(2008).

    [126] Annibale P, Vanni S, Scarselli M et al. Quantitative photo activated localization microscopy: unraveling the effects of photoblinking[J]. PloS One, 6, e22678(2011).

    [127] Huang B, Wang W, Bates M et al. Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy[J]. Science, 319, 810-813(2008).

    [128] Yang J Y, Dong H, Xing F L et al. Single-molecule localization super-resolution microscopy and its applications[J]. Laser & Optoelectronics Progress, 58, 1200001(2021).

    [129] Zuo C, Feng S J, Zhang X Y et al. Deep learning based computational imaging: status, challenges, and future[J]. Acta Optica Sinica, 40, 0111003(2020).

    [130] Wang F, Wang H, Bian Y M et al. Applications of deep learning in computational imaging[J]. Acta Optica Sinica, 40, 0111002(2020).

    [131] Shao X P, Liu F, Li W et al. Latest progress in computational imaging technology and application[J]. Laser & Optoelectronics Progress, 57, 020001(2020).

    [133] Rahman M S S, Li J, Mengu D et al. Ensemble learning of diffractive optical networks[J]. Light, Science & Applications, 10, 14(2021).

    [134] Zhou T K, Lin X, Wu J M et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit[J]. Nature Photonics, 15, 367-373(2021).

    [135] Redmon J, Farhadi A. YOLO9000: better, faster, stronger[C]. //2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017. Honolulu, HI, 6517-6525(2017).

    [136] Litjens G, Kooi T, Bejnordi B E et al. A survey on deep learning in medical image analysis[J]. Medical Image Analysis, 42, 60-88(2017).

    [137] Schmidt U, Weigert M, Broaddus C et al. Cell detection with star-convex polygons[M]. //Frangi A F, Schnabel J A, Davatzikos C, et al. Medical image computing and computer assisted intervention-MICCAI 2018. Lecture notes in computer science, 11071, 265-273(2018).

    [138] Weigert M, Schmidt U, Haase R et al. Star-convex polyhedra for 3D object detection and segmentation in microscopy[C]. //2020 IEEE Winter Conference on Applications of Computer Vision (WACV), March 1-5, 2020, Snowmass, CO, USA., 3655-3662(2020).

    [139] Krull A, Buchholz T O, Jug F. Noise2Void-learning denoising from single noisy images[C]. //2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 15-20, 2019, Long Beach, CA, USA., 2124-2132(2019).

    [140] Araújo T, Aresta G, Castro E et al. Classification of breast cancer histology images using convolutional neural networks[J]. PLoS One, 12, e0177544(2017).

    [141] von Chamier L, Laine R F, Jukkala J et al. Democratising deep learning for microscopy with ZeroCostDL4Mic[J]. Nature Communications, 12, 2276(2021).

    Tools

    Get Citation

    Copy Citation Text

    Haoyu Li, Liying Qu, Zijie Hua, Xinwei Wang, Weisong Zhao, Jian Liu. Deep Learning Based Fluorescence Microscopy Imaging Technologies and Applications[J]. Laser & Optoelectronics Progress, 2021, 58(18): 1811007

    Download Citation

    EndNote(RIS)BibTexPlain Text
    Save article for my favorites
    Paper Information

    Category: Imaging Systems

    Received: Jun. 1, 2021

    Accepted: Aug. 9, 2021

    Published Online: Sep. 3, 2021

    The Author Email: Liu Jian (liujian@hit.edu.cn)

    DOI:10.3788/LOP202158.1811007

    Topics