Laser & Optoelectronics Progress, Volume. 58, Issue 18, 1811007(2021)
Deep Learning Based Fluorescence Microscopy Imaging Technologies and Applications
[1] Lichtman J W, Conchello J A. Fluorescence microscopy[J]. Nature Methods, 2, 910-919(2005).
[2] Huisken J, Swoger J, del Bene F et al. Optical sectioning deep inside live embryos by selective plane illumination microscopy[J]. Science, 305, 1007-1009(2004).
[3] Chen B C, Legant W R, Wang K et al. Lattice light-sheet microscopy: imaging molecules to embryos at high spatiotemporal resolution[J]. Science, 346, 1257998(2014).
[5] Gustafsson M G. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy[J]. Journal of Microscopy, 198, 82-87(2000).
[6] Betzig E, Patterson G H, Sougrat R et al. Imaging intracellular fluorescent proteins at nanometer resolution[J]. Science, 313, 1642-1645(2006).
[7] Rust M J, Bates M, Zhuang X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM)[J]. Nature Methods, 3, 793-795(2006).
[8] Hell S W, Wichmann J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy[J]. Optics Letters, 19, 780-782(1994).
[9] Sigal Y M, Zhou R B, Zhuang X W. Visualizing and discovering cellular structures with super-resolution microscopy[J]. Science, 361, 880-887(2018).
[10] Nixon-Abell J, Obara C J, Weigel A V et al. Increased spatiotemporal resolution reveals highly dynamic dense tubular matrices in the peripheral ER[J]. Science, 354, aaf3928(2016).
[11] Valm A M, Cohen S, Legant W R et al. Applying systems-level spectral imaging and analysis to reveal the organelle interactome[J]. Nature, 546, 162-167(2017).
[12] Hess S T, Gould T J, Gudheti M V et al. Dynamic clustered distribution of hemagglutinin resolved at 40 nm in living cell membranes discriminates between raft theories[J]. Proceedings of the National Academy of Sciences of USA, 104, 17370-17375(2007).
[13] Shroff H, Galbraith C G, Galbraith J A et al. Live-cell photoactivated localization microscopy of nanoscale adhesion dynamics[J]. Nature Methods, 5, 417-423(2008).
[14] Westphal V, Rizzoli S O, Lauterbach M A et al. Video-rate far-field optical nanoscopy dissects synaptic vesicle movement[J]. Science, 320, 246-249(2008).
[15] Shin W, Ge L H, Arpino G et al. Visualization of membrane pore in live cells reveals a dynamic-pore theory governing fusion and endocytosis[J]. Cell, 173, 934-945(2018).
[17] Huang X, Fan J, Li L et al. Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy[J]. Nature Biotechnology, 36, 451-459(2018).
[18] Li D, Shao L, Chen B C et al. Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics[J]. Science, 349, aab3500(2015).
[19] Guo Y T, Li D, Zhang S W et al. Visualizing intracellular organelle and cytoskeletal interactions at nanoscale resolution on millisecond timescales[J]. Cell, 175, 1430-1442(2018).
[20] Jacquemet G, Carisey A F, Hamidi H et al. The cell biologist’s guide to super-resolution microscopy[J]. Journal of Cell Science, 133, jcs.240713(2020).
[21] Icha J, Weber M, Waters J C et al. Phototoxicity in live fluorescence microscopy, and how to avoid it[J]. BioEssays, 39, 1700003(2017).
[22] Laissue P P, Alghamdi R A, Tomancak P et al. Assessing phototoxicity in live fluorescence imaging[J]. Nature Methods, 14, 657-661(2017).
[23] Mortensen K I, Churchman L S, Spudich J A et al. Optimized localization analysis for single-molecule tracking and super-resolution microscopy[J]. Nature Methods, 7, 377-381(2010).
[24] Dempsey G T, Vaughan J C, Chen K H et al. Evaluation of fluorophores for optimal performance in localization-based super-resolution imaging[J]. Nature Methods, 8, 1027-1036(2011).
[25] Helmerich D A, Beliu G, Matikonda S S et al. Photoblueing of organic dyes can cause artifacts in super-resolution microscopy[J]. Nature Methods, 18, 253-257(2021).
[26] Erdélyi M, Sinkó J, Kákonyi R et al. Origin and compensation of imaging artefacts in localization-based super-resolution microscopy[J]. Methods, 88, 122-132(2015).
[27] Schaefer L H, Schuster D, Schaffer J. Structured illumination microscopy: artefact analysis and reduction utilizing a parameter optimization approach[J]. Journal of Microscopy, 216, 165-174(2004).
[28] Demmerle J, Innocent C, North A J et al. Strategic and practical guidelines for successful structured illumination microscopy[J]. Nature Protocols, 12, 988-1010(2017).
[29] Mo Y, Feng F, Mao H et al. Structured illumination microscopy artefacts caused by illumination scattering[J]. Philosophical Transactions of the Royal Society A, 379, 20200153(2021).
[30] Gwosch K C, Pape J K, Balzarotti F et al. MINFLUX nanoscopy delivers 3D multicolor nanometer resolution in cells[J]. Nature Methods, 17, 217-224(2020).
[31] Bodén A, Pennacchietti F, Coceano G et al. Volumetric live cell imaging with three-dimensional parallelized RESOLFT microscopy[J]. Nature Biotechnology, 39, 609-618(2021).
[32] Hao X, Allgeyer E S, Lee D R et al. Three-dimensional adaptive optical nanoscopy for thick specimen imaging at sub-50-nm resolution[J]. Nature Methods, 18, 688-693(2021).
[33] Liu T L, Upadhyayula S, Milkie D E et al. Observing the cell in its native state: imaging subcellular dynamics in multicellular organisms[J]. Science, 360, 1-15(2018).
[34] Lin R Z, Kipreos E T, Zhu J et al. Subcellular three-dimensional imaging deep through multicellular thick samples by structured illumination microscopy and adaptive optics[J]. Nature Communications, 12, 3148(2021).
[35] Mahecic D, Gambarotto D, Douglass K M et al. Homogeneous multifocal excitation for high-throughput super-resolution imaging[J]. Nature Methods, 17, 726-733(2020).
[36] Douglass K M, Sieben C, Archetti A et al. Super-resolution imaging of multiple cells by optimized flat-field epi-illumination[J]. Nature Photonics, 10, 705-708(2016).
[37] Yu J Y, Becker S R, Folberth J et al. Achieving superresolution with illumination-enhanced sparsity[J]. Optics Express, 26, 9850-9865(2018).
[38] Tenne R, Rossman U, Rephael B et al. Super-resolution enhancement by quantum image scanning microscopy[J]. Nature Photonics, 13, 116-122(2019).
[39] Masullo L A, Bodén A, Pennacchietti F et al. Enhanced photon collection enables four dimensional fluorescence nanoscopy of living systems[J]. Nature Communications, 9, 3281(2018).
[40] Dreier J, Castello M, Coceano G et al. Smart scanning for low-illumination and fast RESOLFT nanoscopy in vivo[J]. Nature Communications, 10, 556(2019).
[41] Zhang Y, Schroeder L K, Lessard M D et al. Nanoscale subcellular architecture revealed by multicolor three-dimensional salvaged fluorescence imaging[J]. Nature Methods, 17, 225-231(2020).
[42] Zhang Z, Kenny S J, Hauser M et al. Ultrahigh-throughput single-molecule spectroscopy and spectrally resolved super-resolution microscopy[J]. Nature Methods, 12, 935-938(2015).
[43] Belthangady C, Royer L A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction[J]. Nature Methods, 16, 1215-1225(2019).
[44] Gustafsson N, Culley S, Ashdown G et al. Fast live-cell conventional fluorophore nanoscopy with ImageJ through super-resolution radial fluctuations[J]. Nature Communications, 7, 12471(2016).
[45] Dertinger T, Colyer R, Iyer G et al. Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI)[J]. Proceedings of the National Academy of Sciences of USA, 106, 22287-22292(2009).
[46] Agarwal K, Macháň R. Multiple signal classification algorithm for super-resolution fluorescence microscopy[J]. Nature Communications, 7, 13752(2016).
[47] Cox S, Rosten E, Monypenny J et al. Bayesian localization microscopy reveals nanoscale podosome dynamics[J]. Nature Methods, 9, 195-200(2011).
[49] Yahiatene I, Hennig S, Müller M et al. Entropy-based super-resolution imaging (ESI): from disorder to fine detail[J]. ACS Photonics, 2, 1049-1056(2015).
[50] Richardson W H. Bayesian-based iterative method of image restoration[J]. Journal of the Optical Society of America, 62, 55-59(1972).
[51] Arigovindan M, Fung J C, Elnatan D et al. High-resolution restoration of 3D structures from widefield images with extreme low signal-to-noise-ratio[J]. Proceedings of the National Academy of Sciences of the USA, 110, 17344-17349(2013).
[54] Aspelmeier T, Egner A, Munk A. Modern statistical challenges in high-resolution fluorescence microscopy[J]. Annual Review of Statistics and Its Application, 2, 163-202(2015).
[55] Blasse C, Saalfeld S, Etournay R et al. PreMosa: extracting 2D surfaces from 3D microscopy mosaics[J]. Bioinformatics, 33, 2563-2569(2017).
[56] Shihavuddin A, Basu S, Rexhepaj E et al. Smooth 2D manifold extraction from 3D image stack[J]. Nature Communications, 8, 15554(2017).
[57] LeCun Y, Bengio Y, Hinton G. Deep learning[J]. Nature, 521, 436-444(2015).
[58] Waller L, Tian L. Machine learning for 3D microscopy[J]. Nature, 523, 416-417(2015).
[59] Cortes C, Vapnik V. Support-vector networks[J]. Machine Learning, 20, 273-297(1995).
[60] Reynolds D A, Quatieri T F, Dunn R B. Speaker verification using adapted Gaussian mixture models[J]. Digital Signal Processing, 10, 19-41(2000).
[61] Alam F I, Zhou J, Liew A W C et al. Conditional random field and deep feature learning for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 57, 1612-1628(2019).
[62] Rabiner L R. A tutorial on hidden Markov models and selected applications in speech recognition[J]. Proceedings of the IEEE, 77, 257-286(1989).
[63] Hosmer D W, Jr, Lemeshow S, Sturdivant R X. Applied logistic regression[M](2013).
[64] McCulloch W S, Pitts W. A logical calculus of the ideas immanent in nervous activity[J]. The Bulletin of Mathematical Biophysics, 5, 115-133(1943).
[65] Turing A M. Computing machinery and intelligence[M]. //Epstein R, Roberts G, Beber G. Parsing the Turing test, 23-65(2007).
[66] Rosenblatt F. The perceptron: a probabilistic model for information storage and organization in the brain[J]. Psychological Review, 65, 386-408(1958).
[67] Widrow B. Adaptive sampled-data systems[J]. IFAC Proceedings Volumes, 1, 433-439(1960).
[68] Minsky M, Papert S A. Perceptrons: an introduction to computational geometry[M](2017).
[69] Werbos P. Beyond regression: new tools for prediction and analysis in the behavioral sciences[D](1974).
[70] Fukushima K, Miyake S. Neocognitron: a self-organizing neural network model for a mechanism of visual pattern recognition[M]. //Amari S I, Arbib M A. Competition and cooperation in neural nets. Lecture notes in computer science, 45, 267-285(1982).
[71] Kohonen T. Self-organized formation of topologically correct feature maps[J]. Biological Cybernetics, 43, 59-69(1982).
[72] Hopfield J J. Neural networks and physical systems with emergent collective computational abilities[J]. Proceedings of the National Academy of Sciences of the USA, 79, 2554-2558(1982).
[73] Hinton G E, Sejnowski T J. Learning and relearning in Boltzmann machines[J]. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, 1, 282-317(1986).
[74] Rumelhart D E, McClelland J L. PDP Research Group. Parallel distributed processing[M](1986).
[75] Jordan M I. Serial order: a parallel distributed processing approach[J]. Advances in Psychology, 121, 471-495(1997).
[76] Rumelhart D E, Hinton G E, Williams R J. Learning internal representations by error propagation[M]. //Readings in cognitive science: a perspective from psychology and artificial intelligence, 399-421(1988).
[77] LeCun Y, Bottou L, Bengio Y et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 86, 2278-2324(1998).
[78] Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural Computation, 9, 1735-1780(1997).
[79] Schuster M, Paliwal K K. Bidirectional recurrent neural networks[J]. IEEE Transactions on Signal Processing, 45, 2673-2681(1997).
[80] Hinton G E, Osindero S, Teh Y W. A fast learning algorithm for deep belief nets[J]. Neural Computation, 18, 1527-1554(2006).
[81] Salakhutdinov R, Hinton G E. Deep Boltzmann machines[J]. Journal of Machine Learning Research, 5, 448-455(2009).
[82] Nair V, Hinton G E. Rectified linear units improve restricted Boltzmann machines[C]. //Proceedings of the 27th Annual International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel. [S.l.:s.n.], 807-814(2010).
[85] Silver D, Huang A, Maddison C J et al. Mastering the game of Go with deep neural networks and tree search[J]. Nature, 529, 484-489(2016).
[87] Bengio Y. Deep learning of representations: looking forward[C]. //International Conference on Statistical Language and Speech Processing (SLSP), July 29-31, 2013, Tarragona, Spain, 1-37(2013).
[88] Deng L, Yu D. Deep learning: methods and applications[J]. Foundations and Trends ® in Signal Processing, 7, 197-387(2014).
[89] Szepesvári C. Algorithms for reinforcement learning[J]. Synthesis Lectures on Artificial Intelligence and Machine Learning, 4, 1-103(2010).
[90] Bai C, Liu C, Yu X H et al. Imaging enhancement of light-sheet fluorescence microscopy via deep learning[J]. IEEE Photonics Technology Letters, 31, 1803-1806(2019).
[91] Zhang H, Zhao Y X, Fang C Y et al. Exceeding the limits of 3D fluorescence microscopy using a dual-stage-processing network[J]. Optica, 7, 1627-1640(2020).
[92] Wagner N, Beuttenmueller F, Norlin N et al. Deep learning-enhanced light-field imaging with continuous validation[J]. Nature Methods, 18, 557-563(2021).
[93] Ling C, Zhang C L, Wang M Q et al. Fast structured illumination microscopy via deep learning[J]. Photonics Research, 8, 1350-1359(2020).
[95] Nehme E, Weiss L E, Michaeli T et al. Deep-STORM: super-resolution single-molecule microscopy by deep learning[J]. Optica, 5, 458-464(2018).
[96] Nehme E, Freedman D, Gordon R et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning[J]. Nature Methods, 17, 734-740(2020).
[97] Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation[M]. //Navab N, Hornegger J, Wells W M, et al.Medical image computing and computer-assisted intervention-MICCAI 2015. Lecture notes in computer science, 9351, 234-241(2015).
[98] Weigert M, Schmidt U, Boothe T et al. Content-aware image restoration: pushing the limits of fluorescence microscopy[J]. Nature Methods, 15, 1090-1097(2018).
[99] Fang L, Monroe F, Novak S W et al. Deep learning-based point-scanning super-resolution imaging[J]. Nature Methods, 18, 406-416(2021).
[100] Zhao F, Zhu L X, Fang C Y et al. Deep-learning super-resolution light-sheet add-on microscopy (Deep-SLAM) for easy isotropic volumetric imaging of large biological specimens[J]. Biomedical Optics Express, 11, 7273-7285(2020).
[101] Wang Z, Zhu L, Zhang H et al. Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning[J]. Nature Methods, 18, 551-556(2021).
[102] Jin L, Liu B, Zhao F et al. Deep learning enables structured illumination microscopy with low light levels and enhanced speed[J]. Nature Communications, 11, 1934(2020).
[103] Ouyang W, Aristov A, Lelek M et al. Deep learning massively accelerates super-resolution localization microscopy[J]. Nature Biotechnology, 36, 460-468(2018).
[106] Li X, Zhang G, Qiao H et al. Unsupervised content-preserving transformation for optical microscopy[J]. Light, Science & Applications, 10, 44(2021).
[107] Li X, Dong J Y, Li B W et al. Fast confocal microscopy imaging based on deep learning[C]. //2020 IEEE International Conference on Computational Photography (ICCP), April 24-26, 2020, St. Louis, MO, USA., 1-12(2020).
[108] Wang H D, Rivenson Y, Jin Y Y et al. Deep-learning enables cross-modality super-resolution in fluorescence microscopy[J]. Nature Methods, 16, 103-110(2019).
[109] Ling C, Du L P, Yuan X C. Deep learning approach for nonlinear structured illumination microscopy[J]. Proceedings of SPIE, 11565, 115650P(2020).
[110] Qiao C, Li D, Guo Y et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy[J]. Nature Methods, 18, 194-202(2021).
[111] Li M Z, Shan H M, Pryshchep S et al. Deep adversarial network for super stimulated emission depletion imaging[J]. Journal of Nanophotonics, 14, 016009(2020).
[112] Zhang Y L, Li K P, Li K et al. Image super-resolution using very deep residual channel attention networks[M]. //Ferrari V, Hebert M, Sminchisescu C, et al. Computer vision-ECCV 2018. Lecture notes in computer science, 11211, 294-310(2018).
[113] Vaswani A, Shazeer N, Parmar N et al. Attention is all you need[C]. //Advances in Neural Information Processing Systems(NIPS), December 4-9, 2017, Long Beach, Canada, 5998-6008(2017).
[114] Christensen C N, Ward E N, Lu M et al. ML-SIM: universal reconstruction of structured illumination microscopy images using transfer learning[J]. Biomedical Optics Express, 12, 2720-2733(2021).
[115] Rivenson Y, Göröcs Z, Günaydin H et al. Deep learning microscopy[J]. Optica, 4, 1437-1443(2017).
[116] Jonkman J, Brown C M, Wright G D et al. Tutorial: guidance for quantitative confocal microscopy[J]. Nature Protocols, 15, 1585-1611(2020).
[117] Santi P A. Light sheet fluorescence microscopy: a review[J]. The Journal of Histochemistry and Cytochemistry, 59, 129-138(2011).
[118] Yu X H, Liu C, Bai C et al. Progress in light-sheet fluorescence microscopy and applications[J]. Laser & Optoelectronics Progress, 57, 100001(2020).
[119] Levoy M, Ng R, Adams A et al. Light field microscopy[C]. //ACM SIGGRAPH 2006 Papers on-SIGGRAPH’06, July 30-August 3, 2006. Boston, Massachusetts., 924-934(2006).
[121] Saxena M, Eluru G, Gorthi S S. Structured illumination microscopy[J]. Advances in Optics and Photonics, 7, 241-275(2015).
[122] Schermelleh L, Ferrand A, Huser T et al. Super-resolution microscopy demystified[J]. Nature Cell Biology, 21, 72-84(2019).
[123] Wang Y, Zhao W, Wang K et al. Fluorescent nanoparticles for stimulated emission depletion microscopy[J]. Journal of Nanoscience and Nanotechnology, 20, 2308-2315(2020).
[125] Manley S, Gillette J M, Patterson G H et al. High-density mapping of single-molecule trajectories with photoactivated localization microscopy[J]. Nature Methods, 5, 155-157(2008).
[126] Annibale P, Vanni S, Scarselli M et al. Quantitative photo activated localization microscopy: unraveling the effects of photoblinking[J]. PloS One, 6, e22678(2011).
[127] Huang B, Wang W, Bates M et al. Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy[J]. Science, 319, 810-813(2008).
[128] Yang J Y, Dong H, Xing F L et al. Single-molecule localization super-resolution microscopy and its applications[J]. Laser & Optoelectronics Progress, 58, 1200001(2021).
[129] Zuo C, Feng S J, Zhang X Y et al. Deep learning based computational imaging: status, challenges, and future[J]. Acta Optica Sinica, 40, 0111003(2020).
[130] Wang F, Wang H, Bian Y M et al. Applications of deep learning in computational imaging[J]. Acta Optica Sinica, 40, 0111002(2020).
[131] Shao X P, Liu F, Li W et al. Latest progress in computational imaging technology and application[J]. Laser & Optoelectronics Progress, 57, 020001(2020).
[133] Rahman M S S, Li J, Mengu D et al. Ensemble learning of diffractive optical networks[J]. Light, Science & Applications, 10, 14(2021).
[134] Zhou T K, Lin X, Wu J M et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit[J]. Nature Photonics, 15, 367-373(2021).
[135] Redmon J, Farhadi A. YOLO9000: better, faster, stronger[C]. //2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017. Honolulu, HI, 6517-6525(2017).
[136] Litjens G, Kooi T, Bejnordi B E et al. A survey on deep learning in medical image analysis[J]. Medical Image Analysis, 42, 60-88(2017).
[137] Schmidt U, Weigert M, Broaddus C et al. Cell detection with star-convex polygons[M]. //Frangi A F, Schnabel J A, Davatzikos C, et al. Medical image computing and computer assisted intervention-MICCAI 2018. Lecture notes in computer science, 11071, 265-273(2018).
[138] Weigert M, Schmidt U, Haase R et al. Star-convex polyhedra for 3D object detection and segmentation in microscopy[C]. //2020 IEEE Winter Conference on Applications of Computer Vision (WACV), March 1-5, 2020, Snowmass, CO, USA., 3655-3662(2020).
[139] Krull A, Buchholz T O, Jug F. Noise2Void-learning denoising from single noisy images[C]. //2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 15-20, 2019, Long Beach, CA, USA., 2124-2132(2019).
[140] Araújo T, Aresta G, Castro E et al. Classification of breast cancer histology images using convolutional neural networks[J]. PLoS One, 12, e0177544(2017).
[141] von Chamier L, Laine R F, Jukkala J et al. Democratising deep learning for microscopy with ZeroCostDL4Mic[J]. Nature Communications, 12, 2276(2021).
Get Citation
Copy Citation Text
Haoyu Li, Liying Qu, Zijie Hua, Xinwei Wang, Weisong Zhao, Jian Liu. Deep Learning Based Fluorescence Microscopy Imaging Technologies and Applications[J]. Laser & Optoelectronics Progress, 2021, 58(18): 1811007
Category: Imaging Systems
Received: Jun. 1, 2021
Accepted: Aug. 9, 2021
Published Online: Sep. 3, 2021
The Author Email: Liu Jian (liujian@hit.edu.cn)