Chinese Journal of Lasers, Volume. 49, Issue 24, 2407206(2022)
Deep Learning in Single-Molecule Localization Microscopy
[1] Zhuang X W. Nano-imaging with STORM[J]. Nature Photonics, 3, 365-367(2009).
[2] Abbe E. Note on the proper definition of the amplifying power of a lens or lens-system[J]. Journal of the Royal Microscopical Society, 4, 348-351(1884).
[3] Gustafsson M G L. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy[J]. Journal of Microscopy, 198, 82-87(2000).
[4] Heintzmann R, Jovin T M, Cremer C. Saturated patterned excitation microscopy: a concept for optical resolution improvement[J]. Journal of the Optical Society of America A, 19, 1599-1609(2002).
[5] Hell S W, Wichmann J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy[J]. Optics Letters, 19, 780-782(1994).
[6] Betzig E, Patterson G H, Sougrat R et al. Imaging intracellular fluorescent proteins at nanometer resolution[J]. Science, 313, 1642-1645(2006).
[7] Hess S T, Girirajan T P K, Mason M D. Ultra-high resolution imaging by fluorescence photoactivation localization microscopy[J]. Biophysical Journal, 91, 4258-4272(2006).
[8] Rust M J, Bates M, Zhuang X W. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM)[J]. Nature Methods, 3, 793-796(2006).
[9] Nehme E, Weiss L E, Michaeli T et al. Deep-STORM: super-resolution single-molecule microscopy by deep learning[J]. Optica, 5, 458-464(2018).
[10] Kim T, Moon S, Xu K. Information-rich localization microscopy through machine learning[J]. Nature Communications, 10, 1-8(2019).
[11] Ouyang W, Aristov A, Lelek M et al. Deep learning massively accelerates super-resolution localization microscopy[J]. Nature Biotechnology, 36, 460-468(2018).
[12] Huang B, Wang W Q, Bates M et al. Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy[J]. Science, 319, 810-813(2008).
[13] Shroff H, Galbraith C G, Galbraith J A et al. Dual-color superresolution imaging of genetically expressed probes within individual adhesion complexes[J]. Proceedings of the National Academy of Sciences of the United States of America, 104, 20308-20313(2007).
[14] Subach F V, Patterson G H, Manley S et al. Photoactivatable mCherry for high-resolution two-color fluorescence microscopy[J]. Nature Methods, 6, 153-159(2009).
[15] Xu K, Zhong G S, Zhuang X W. Actin, spectrin, and associated proteins form a periodic cytoskeletal structure in axons[J]. Science, 339, 452-456(2013).
[16] Dempsey G T, Vaughan J C, Chen K H et al. Evaluation of fluorophores for optimal performance in localization-based super-resolution imaging[J]. Nature Methods, 8, 1027-1036(2011).
[17] Zhang Z Y, Kenny S J, Hauser M et al. Ultrahigh-throughput single-molecule spectroscopy and spectrally resolved super-resolution microscopy[J]. Nature Methods, 12, 935-938(2015).
[18] Jameson D M, Ross J A. Fluorescence polarization/anisotropy in diagnostics and imaging[J]. Chemical Reviews, 110, 2685-2708(2010).
[19] Backlund M P, Lew M D, Backer A S et al. The role of molecular dipole orientation in single-molecule fluorescence microscopy and implications for super-resolution imaging[J]. Chemphyschem, 15, 587-599(2014).
[20] Shroder D Y, Lippert L G, Goldman Y E. Single molecule optical measurements of orientation and rotations of biological macromolecules[J]. Methods and Applications in Fluorescence, 4, 042004(2016).
[21] Manzo C, Garcia-Parajo M F. A review of progress in single particle tracking: from methods to biophysical insights[J]. Reports on Progress in Physics, 78, 124601(2015).
[22] Kusumi A, Tsunoyama T A, Hirosawa K M et al. Tracking single molecules at work in living cells[J]. Nature Chemical Biology, 10, 524-532(2014).
[23] Berezin M Y, Achilefu S. Fluorescence lifetime measurements and biological imaging[J]. Chemical Reviews, 110, 2641-2684(2010).
[24] Datta R, Heaster T M, Sharick J T et al. Fluorescence lifetime imaging microscopy: fundamentals and advances in instrumentation, analysis, and applications[J]. Journal of Biomedical Optics, 25, 071203(2020).
[25] Pavani S R P, Thompson M A, Biteen J S et al. Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function[J]. Proceedings of the National Academy of Sciences of the United States of America, 106, 2995-2999(2009).
[26] Shechtman Y, Sahl S J, Backer A S et al. Optimal point spread function design for 3D imaging[J]. Physical Review Letters, 113, 133902(2014).
[27] Juette M F, Gould T J, Lessard M D et al. Three-dimensional sub-100 nm resolution fluorescence microscopy of thick samples[J]. Nature Methods, 5, 527-529(2008).
[28] Babcock H P. Multiplane and spectrally-resolved single molecule localization microscopy with industrial grade CMOS cameras[J]. Scientific Reports, 8, 1726(2018).
[29] Cabriel C, Bourg N, Jouchet P et al. Combining 3D single molecule localization strategies for reproducible bioimaging[J]. Nature Communications, 10, 1980(2019).
[30] Dasgupta A, Deschamps J, Matti U et al. Direct supercritical angle localization microscopy for nanometer 3D superresolution[J]. Nature Communications, 12, 1180(2021).
[31] Shechtman Y, Weiss L E, Backer A S et al. Multicolour localization microscopy by point-spread-function engineering[J]. Nature Photonics, 10, 590-594(2016).
[32] Manley S, Gillette J M, Patterson G H et al. High-density mapping of single-molecule trajectories with photoactivated localization microscopy[J]. Nature Methods, 5, 155-157(2008).
[33] Shim S H, Xia C L, Zhong G S et al. Super-resolution fluorescence imaging of organelles in live cells with photoswitchable membrane probes[J]. Proceedings of the National Academy of Sciences of the United States of America, 109, 13978-13983(2012).
[34] Mohan N, Sorokina E M, Verdeny I V et al. Detyrosinated microtubules spatially constrain lysosomes facilitating lysosome-autophagosome fusion[J]. The Journal of Cell Biology, 218, 632-643(2019).
[35] Otterstrom J, Castells-Garcia A, Vicario C et al. Super-resolution microscopy reveals how histone tail acetylation affects DNA compaction within nucleosomes in vivo[J]. Nucleic Acids Research, 47, 8470-8484(2019).
[36] Bálint Š, Verdeny Vilanova I, Sandoval Álvarez Á et al. Correlative live-cell and superresolution microscopy reveals cargo transport dynamics at microtubule intersections[J]. Proceedings of the National Academy of Sciences of the United States of America, 110, 3375-3380(2013).
[37] Doksani Y, Wu J Y, de Lange T et al. Super-resolution fluorescence imaging of telomeres reveals TRF2-dependent T-loop formation[J]. Cell, 155, 345-356(2013).
[38] LeCun Y, Bengio Y, Hinton G. Deep learning[J]. Nature, 521, 436-444(2015).
[39] Leshno M, Lin V Y, Pinkus A et al. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function[J]. Neural Networks, 6, 861-867(1993).
[40] Lu Z, Pu H M, Wang F C et al. The expressive power of neural networks: a view from the width[C], 6232-6240(2017).
[41] Cunningham P, Cord M, Delany S J. Supervised learning[M]. Cord M, Cunningham P.Machine learning techniques for multimedia. Cognitive technologies, 21-49(2008).
[42] Barlow H B. Unsupervised learning[J]. Neural Computation, 1, 295-311(1989).
[44] Rumelhart D E, Hinton G E, Williams R J. Learning representations by back-propagating errors[J]. Nature, 323, 533-536(1986).
[45] Voulodimos A, Doulamis N, Doulamis A et al. Deep learning for computer vision: a brief review[J]. Computational Intelligence and Neuroscience, 2018, 7068349(2018).
[46] Steinkraus D, Buck I, Simard P Y. Using GPUs for machine learning algorithms[C], 1115-1120(2005).
[47] Raina R, Madhavan A, Ng A Y. Large-scale deep unsupervised learning using graphics processors[C], 873-880(2009).
[48] Widrow B, Lehr M A. 30 years of adaptive neural networks: perceptron, madaline, and backpropagation[J]. Proceedings of the IEEE, 78, 1415-1442(1990).
[49] Zeiler M D, Fergus R. Visualizing and understanding convolutional networks[M]. Fleet D, Pajdla T, Schiele B, et al.Computervision-ECCV2014. Lecture Notes in Computer Science, 8689, 818-833(2014).
[50] Noh H, Hong S, Han B. Learning deconvolution network for semantic segmentation[C], 1520-1528(2015).
[51] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 60, 84-90(2017).
[52] LeCun Y, Bottou L, Bengio Y et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 86, 2278-2324(1998).
[54] Szegedy C, Liu W, Jia Y Q et al. Going deeper with convolutions[C](2015).
[55] Szegedy C, Vanhoucke V, Ioffe S et al. Rethinking the inception architecture for computer vision[C], 2818-2826(2016).
[56] He K M, Zhang X Y, Ren S Q et al. Deep residual learning for image recognition[C], 770-778(2016).
[57] Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C], 7132-7141(2018).
[58] Szegedy C, Ioffe S, Vanhoucke V et al. Inception-v4, inception-ResNet and the impact of residual connections on learning[C], 4278-4284(2017).
[59] Badrinarayanan V, Kendall A, Cipolla R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 2481-2495(2017).
[61] Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation[C], 3431-3440(2015).
[62] Newell A, Yang K Y, Deng J. Stacked hourglass networks for human pose estimation[M]. Leibe B, Matas J, Sebe N, et al.Computervision-ECCV2016. Lecture Notes in Computer Science, 9912, 483-499(2016).
[63] Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation[M]. Navab N, Hornegger J, Wells W M, et al.Medical image computing and computer-assisted intervention-MICCAI 2015. Lecture Notes in Computer Science, 9351, 234-241(2015).
[64] Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural Computation, 9, 1735-1780(1997).
[72] Zhu J Y, Park T, Isola P et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C], 2242-2251(2017).
[73] Karras T, Laine S, Aila T M. A style-based generator architecture for generative adversarial networks[C], 4396-4405(2019).
[74] Zhang P Y, Liu S, Chaurasia A et al. Analyzing complex single-molecule emission patterns with deep learning[J]. Nature Methods, 15, 913-916(2018).
[76] Zelger P, Kaser K, Rossboth B et al. Three-dimensional localization microscopy using deep learning[J]. Optics Express, 26, 33166-33179(2018).
[77] Nehme E, Freedman D, Gordon R et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning[J]. Nature Methods, 17, 734-740(2020).
[78] Speiser A, Müller L R, Hoess P et al. Deep learning enables fast and dense single-molecule localization with high accuracy[J]. Nature Methods, 18, 1082-1090(2021).
[79] Gaire S K, Zhang Y, Li H Y et al. Accelerating multicolor spectroscopic single-molecule localization microscopy using deep learning[J]. Biomedical Optics Express, 11, 2705-2721(2020).
[80] Hershko E, Weiss L E, Michaeli T et al. Multicolor localization microscopy and point-spread-function engineering by deep learning[J]. Optics Express, 27, 6158-6183(2019).
[81] Nehme E, Ferdman B, Weiss L E et al. Learning optimal wavefront shaping for multi-channel imaging[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43, 2179-2192(2021).
[82] Holden S J, Uphoff S, Kapanidis A N. DAOSTORM: an algorithm for high- density super-resolution microscopy[J]. Nature Methods, 8, 279-280(2011).
[83] Ovesný M, KArˇíAzˇek P, Borkovec J et al. ThunderSTORM: a comprehensive ImageJ plug-in for PALM and STORM data analysis and super-resolution imaging[J]. Bioinformatics, 30, 2389-2390(2014).
[84] Isola P, Zhu J Y, Zhou T H et al. Image-to-image translation with conditional adversarial networks[C], 5967-5976(2017).
[85] Zhou Z W, Kuang W B, Wang Z X et al. ResNet-based image inpainting method for enhancing the imaging speed of single molecule localization microscopy[J]. Optics Express, 30, 31766-31784(2022).
[86] Narayanasamy K K, Rahm J V, Tourani S et al. Fast DNA-PAINT imaging using a deep neural network[J]. Nature Communications, 13, 5047(2022).
[87] Zhu M, Zhang L H, Jin L H et al. DNA-PAINT imaging accelerated by machine learning[J]. Frontiers in Chemistry, 10, 864701(2022).
[88] Shechtman Y, Weiss L E, Backer A S et al. Precise three-dimensional scan-free multiple-particle tracking over large axial ranges with tetrapod point spread functions[J]. Nano Letters, 15, 4194-4199(2015).
[89] Kay S M[M]. Fundamentals of statistical signal processing(1993).
[90] Li Y M, Shi W, Liu S et al. Global fitting for high-accuracy multi-channel single-molecule localization[J]. Nature Communications, 13, 3133(2022).
[91] Li Y M, Mund M, Hoess P et al. Real-time 3D single-molecule localization using experimental point spread functions[J]. Nature Methods, 15, 367-369(2018).
[92] Möckl L, Roy A R, Petrov P N et al. Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural network BGnet[J]. Proceedings of the National Academy of Sciences of the United States of America, 117, 60-67(2020).
[93] Zhang Z Y, Zhang Y, Ying L et al. Machine-learning based spectral classification for spectroscopic single-molecule localization microscopy[J]. Optics Letters, 44, 5864-5867(2019).
[94] Jeong D, Kim D. Super-resolution fluorescence microscopy-based single-molecule spectroscopy[J]. Bulletin of the Korean Chemical Society, 43, 316-327(2022).
[95] Khater I M, Nabi I R, Hamarneh G. A review of super-resolution single-molecule localization microscopy cluster analysis and quantification methods[J]. Patterns, 1, 100038(2020).
[96] Sage D, Pham T A, Babcock H et al. Super-resolution fight club: assessment of 2D and 3D single-molecule localization microscopy software[J]. Nature Methods, 16, 387-395(2019).
[97] Venkataramani V, Herrmannsdörfer F, Heilemann M et al. SuReSim: simulating localization microscopy experiments from ground truth models[J]. Nature Methods, 13, 319-321(2016).
[98] Novák T, Gajdos T, Sinkó J et al. TestSTORM: versatile simulator software for multimodal super-resolution localization fluorescence microscopy[J]. Scientific Reports, 7, 951(2017).
[99] Lindén M, ĆGurić V, Boucharin A et al. Simulated single molecule microscopy with SMeagol[J]. Bioinformatics, 32, 2394-2395(2016).
[100] Levet F, Julien G, Galland R et al. A tessellation-based colocalization analysis approach for single-molecule localization microscopy[J]. Nature Communications, 10, 2379(2019).
[101] Culley S, Albrecht D, Jacobs C et al. Quantitative mapping and minimization of super-resolution optical imaging artifacts[J]. Nature Methods, 15, 263-266(2018).
[102] Marsh R J, Pfisterer K, Bennett P et al. Artifact-free high-density localization microscopy analysis[J]. Nature Methods, 15, 689-692(2018).
[103] Spahn C, Herrmannsdörfer F, Kuner T et al. Temporal accumulation analysis provides simplified artifact-free analysis of membrane-protein nanoclusters[J]. Nature Methods, 13, 963-964(2016).
[104] Sieben C, Banterle N, Douglass K M et al. Multicolor single-particle reconstruction of protein complexes[J]. Nature Methods, 15, 777-780(2018).
[105] Cardoen B, Yedder H B, Sharma A et al. ERGO: efficient recurrent graph optimized emitter density estimation in single molecule localization microscopy[J]. IEEE Transactions on Medical Imaging, 39, 1942-1956(2020).
[106] Xu F, Ma D H, MacPherson K P et al. Three-dimensional nanoscopy of whole cells and tissues with in situ point spread function retrieval[J]. Nature Methods, 17, 531-540(2020).
[107] Saguy A, Alalouf O, Opatovski N et al[EB/OL]. DBlink: Dynamic localization microscopy in super spatiotemporal resolution via deep learning.
Get Citation
Copy Citation Text
Tingdan Luo, Yiming Li. Deep Learning in Single-Molecule Localization Microscopy[J]. Chinese Journal of Lasers, 2022, 49(24): 2407206
Category: Optical Diagnostics and Therapy
Received: Aug. 8, 2022
Accepted: Oct. 8, 2022
Published Online: Dec. 19, 2022
The Author Email: Li Yiming (liym2019@sustech.edu.cn)