Chinese Journal of Lasers, Volume. 49, Issue 24, 2407206(2022)

Deep Learning in Single-Molecule Localization Microscopy

Tingdan Luo and Yiming Li*
Author Affiliations
  • Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen 518055, Guangdong, China
  • show less
    References(107)

    [1] Zhuang X W. Nano-imaging with STORM[J]. Nature Photonics, 3, 365-367(2009).

    [2] Abbe E. Note on the proper definition of the amplifying power of a lens or lens-system[J]. Journal of the Royal Microscopical Society, 4, 348-351(1884).

    [3] Gustafsson M G L. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy[J]. Journal of Microscopy, 198, 82-87(2000).

    [4] Heintzmann R, Jovin T M, Cremer C. Saturated patterned excitation microscopy: a concept for optical resolution improvement[J]. Journal of the Optical Society of America A, 19, 1599-1609(2002).

    [5] Hell S W, Wichmann J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy[J]. Optics Letters, 19, 780-782(1994).

    [6] Betzig E, Patterson G H, Sougrat R et al. Imaging intracellular fluorescent proteins at nanometer resolution[J]. Science, 313, 1642-1645(2006).

    [7] Hess S T, Girirajan T P K, Mason M D. Ultra-high resolution imaging by fluorescence photoactivation localization microscopy[J]. Biophysical Journal, 91, 4258-4272(2006).

    [8] Rust M J, Bates M, Zhuang X W. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM)[J]. Nature Methods, 3, 793-796(2006).

    [9] Nehme E, Weiss L E, Michaeli T et al. Deep-STORM: super-resolution single-molecule microscopy by deep learning[J]. Optica, 5, 458-464(2018).

    [10] Kim T, Moon S, Xu K. Information-rich localization microscopy through machine learning[J]. Nature Communications, 10, 1-8(2019).

    [11] Ouyang W, Aristov A, Lelek M et al. Deep learning massively accelerates super-resolution localization microscopy[J]. Nature Biotechnology, 36, 460-468(2018).

    [12] Huang B, Wang W Q, Bates M et al. Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy[J]. Science, 319, 810-813(2008).

    [13] Shroff H, Galbraith C G, Galbraith J A et al. Dual-color superresolution imaging of genetically expressed probes within individual adhesion complexes[J]. Proceedings of the National Academy of Sciences of the United States of America, 104, 20308-20313(2007).

    [14] Subach F V, Patterson G H, Manley S et al. Photoactivatable mCherry for high-resolution two-color fluorescence microscopy[J]. Nature Methods, 6, 153-159(2009).

    [15] Xu K, Zhong G S, Zhuang X W. Actin, spectrin, and associated proteins form a periodic cytoskeletal structure in axons[J]. Science, 339, 452-456(2013).

    [16] Dempsey G T, Vaughan J C, Chen K H et al. Evaluation of fluorophores for optimal performance in localization-based super-resolution imaging[J]. Nature Methods, 8, 1027-1036(2011).

    [17] Zhang Z Y, Kenny S J, Hauser M et al. Ultrahigh-throughput single-molecule spectroscopy and spectrally resolved super-resolution microscopy[J]. Nature Methods, 12, 935-938(2015).

    [18] Jameson D M, Ross J A. Fluorescence polarization/anisotropy in diagnostics and imaging[J]. Chemical Reviews, 110, 2685-2708(2010).

    [19] Backlund M P, Lew M D, Backer A S et al. The role of molecular dipole orientation in single-molecule fluorescence microscopy and implications for super-resolution imaging[J]. Chemphyschem, 15, 587-599(2014).

    [20] Shroder D Y, Lippert L G, Goldman Y E. Single molecule optical measurements of orientation and rotations of biological macromolecules[J]. Methods and Applications in Fluorescence, 4, 042004(2016).

    [21] Manzo C, Garcia-Parajo M F. A review of progress in single particle tracking: from methods to biophysical insights[J]. Reports on Progress in Physics, 78, 124601(2015).

    [22] Kusumi A, Tsunoyama T A, Hirosawa K M et al. Tracking single molecules at work in living cells[J]. Nature Chemical Biology, 10, 524-532(2014).

    [23] Berezin M Y, Achilefu S. Fluorescence lifetime measurements and biological imaging[J]. Chemical Reviews, 110, 2641-2684(2010).

    [24] Datta R, Heaster T M, Sharick J T et al. Fluorescence lifetime imaging microscopy: fundamentals and advances in instrumentation, analysis, and applications[J]. Journal of Biomedical Optics, 25, 071203(2020).

    [25] Pavani S R P, Thompson M A, Biteen J S et al. Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function[J]. Proceedings of the National Academy of Sciences of the United States of America, 106, 2995-2999(2009).

    [26] Shechtman Y, Sahl S J, Backer A S et al. Optimal point spread function design for 3D imaging[J]. Physical Review Letters, 113, 133902(2014).

    [27] Juette M F, Gould T J, Lessard M D et al. Three-dimensional sub-100 nm resolution fluorescence microscopy of thick samples[J]. Nature Methods, 5, 527-529(2008).

    [28] Babcock H P. Multiplane and spectrally-resolved single molecule localization microscopy with industrial grade CMOS cameras[J]. Scientific Reports, 8, 1726(2018).

    [29] Cabriel C, Bourg N, Jouchet P et al. Combining 3D single molecule localization strategies for reproducible bioimaging[J]. Nature Communications, 10, 1980(2019).

    [30] Dasgupta A, Deschamps J, Matti U et al. Direct supercritical angle localization microscopy for nanometer 3D superresolution[J]. Nature Communications, 12, 1180(2021).

    [31] Shechtman Y, Weiss L E, Backer A S et al. Multicolour localization microscopy by point-spread-function engineering[J]. Nature Photonics, 10, 590-594(2016).

    [32] Manley S, Gillette J M, Patterson G H et al. High-density mapping of single-molecule trajectories with photoactivated localization microscopy[J]. Nature Methods, 5, 155-157(2008).

    [33] Shim S H, Xia C L, Zhong G S et al. Super-resolution fluorescence imaging of organelles in live cells with photoswitchable membrane probes[J]. Proceedings of the National Academy of Sciences of the United States of America, 109, 13978-13983(2012).

    [34] Mohan N, Sorokina E M, Verdeny I V et al. Detyrosinated microtubules spatially constrain lysosomes facilitating lysosome-autophagosome fusion[J]. The Journal of Cell Biology, 218, 632-643(2019).

    [35] Otterstrom J, Castells-Garcia A, Vicario C et al. Super-resolution microscopy reveals how histone tail acetylation affects DNA compaction within nucleosomes in vivo[J]. Nucleic Acids Research, 47, 8470-8484(2019).

    [36] Bálint Š, Verdeny Vilanova I, Sandoval Álvarez Á et al. Correlative live-cell and superresolution microscopy reveals cargo transport dynamics at microtubule intersections[J]. Proceedings of the National Academy of Sciences of the United States of America, 110, 3375-3380(2013).

    [37] Doksani Y, Wu J Y, de Lange T et al. Super-resolution fluorescence imaging of telomeres reveals TRF2-dependent T-loop formation[J]. Cell, 155, 345-356(2013).

    [38] LeCun Y, Bengio Y, Hinton G. Deep learning[J]. Nature, 521, 436-444(2015).

    [39] Leshno M, Lin V Y, Pinkus A et al. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function[J]. Neural Networks, 6, 861-867(1993).

    [40] Lu Z, Pu H M, Wang F C et al. The expressive power of neural networks: a view from the width[C], 6232-6240(2017).

    [41] Cunningham P, Cord M, Delany S J. Supervised learning[M]. Cord M, Cunningham P.Machine learning techniques for multimedia. Cognitive technologies, 21-49(2008).

    [42] Barlow H B. Unsupervised learning[J]. Neural Computation, 1, 295-311(1989).

    [44] Rumelhart D E, Hinton G E, Williams R J. Learning representations by back-propagating errors[J]. Nature, 323, 533-536(1986).

    [45] Voulodimos A, Doulamis N, Doulamis A et al. Deep learning for computer vision: a brief review[J]. Computational Intelligence and Neuroscience, 2018, 7068349(2018).

    [46] Steinkraus D, Buck I, Simard P Y. Using GPUs for machine learning algorithms[C], 1115-1120(2005).

    [47] Raina R, Madhavan A, Ng A Y. Large-scale deep unsupervised learning using graphics processors[C], 873-880(2009).

    [48] Widrow B, Lehr M A. 30 years of adaptive neural networks: perceptron, madaline, and backpropagation[J]. Proceedings of the IEEE, 78, 1415-1442(1990).

    [49] Zeiler M D, Fergus R. Visualizing and understanding convolutional networks[M]. Fleet D, Pajdla T, Schiele B, et al.Computervision-ECCV2014. Lecture Notes in Computer Science, 8689, 818-833(2014).

    [50] Noh H, Hong S, Han B. Learning deconvolution network for semantic segmentation[C], 1520-1528(2015).

    [51] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 60, 84-90(2017).

    [52] LeCun Y, Bottou L, Bengio Y et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 86, 2278-2324(1998).

    [54] Szegedy C, Liu W, Jia Y Q et al. Going deeper with convolutions[C](2015).

    [55] Szegedy C, Vanhoucke V, Ioffe S et al. Rethinking the inception architecture for computer vision[C], 2818-2826(2016).

    [56] He K M, Zhang X Y, Ren S Q et al. Deep residual learning for image recognition[C], 770-778(2016).

    [57] Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C], 7132-7141(2018).

    [58] Szegedy C, Ioffe S, Vanhoucke V et al. Inception-v4, inception-ResNet and the impact of residual connections on learning[C], 4278-4284(2017).

    [59] Badrinarayanan V, Kendall A, Cipolla R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 2481-2495(2017).

    [61] Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation[C], 3431-3440(2015).

    [62] Newell A, Yang K Y, Deng J. Stacked hourglass networks for human pose estimation[M]. Leibe B, Matas J, Sebe N, et al.Computervision-ECCV2016. Lecture Notes in Computer Science, 9912, 483-499(2016).

    [63] Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation[M]. Navab N, Hornegger J, Wells W M, et al.Medical image computing and computer-assisted intervention-MICCAI 2015. Lecture Notes in Computer Science, 9351, 234-241(2015).

    [64] Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural Computation, 9, 1735-1780(1997).

    [72] Zhu J Y, Park T, Isola P et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C], 2242-2251(2017).

    [73] Karras T, Laine S, Aila T M. A style-based generator architecture for generative adversarial networks[C], 4396-4405(2019).

    [74] Zhang P Y, Liu S, Chaurasia A et al. Analyzing complex single-molecule emission patterns with deep learning[J]. Nature Methods, 15, 913-916(2018).

    [76] Zelger P, Kaser K, Rossboth B et al. Three-dimensional localization microscopy using deep learning[J]. Optics Express, 26, 33166-33179(2018).

    [77] Nehme E, Freedman D, Gordon R et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning[J]. Nature Methods, 17, 734-740(2020).

    [78] Speiser A, Müller L R, Hoess P et al. Deep learning enables fast and dense single-molecule localization with high accuracy[J]. Nature Methods, 18, 1082-1090(2021).

    [79] Gaire S K, Zhang Y, Li H Y et al. Accelerating multicolor spectroscopic single-molecule localization microscopy using deep learning[J]. Biomedical Optics Express, 11, 2705-2721(2020).

    [80] Hershko E, Weiss L E, Michaeli T et al. Multicolor localization microscopy and point-spread-function engineering by deep learning[J]. Optics Express, 27, 6158-6183(2019).

    [81] Nehme E, Ferdman B, Weiss L E et al. Learning optimal wavefront shaping for multi-channel imaging[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43, 2179-2192(2021).

    [82] Holden S J, Uphoff S, Kapanidis A N. DAOSTORM: an algorithm for high- density super-resolution microscopy[J]. Nature Methods, 8, 279-280(2011).

    [83] Ovesný M, KArˇíAzˇek P, Borkovec J et al. ThunderSTORM: a comprehensive ImageJ plug-in for PALM and STORM data analysis and super-resolution imaging[J]. Bioinformatics, 30, 2389-2390(2014).

    [84] Isola P, Zhu J Y, Zhou T H et al. Image-to-image translation with conditional adversarial networks[C], 5967-5976(2017).

    [85] Zhou Z W, Kuang W B, Wang Z X et al. ResNet-based image inpainting method for enhancing the imaging speed of single molecule localization microscopy[J]. Optics Express, 30, 31766-31784(2022).

    [86] Narayanasamy K K, Rahm J V, Tourani S et al. Fast DNA-PAINT imaging using a deep neural network[J]. Nature Communications, 13, 5047(2022).

    [87] Zhu M, Zhang L H, Jin L H et al. DNA-PAINT imaging accelerated by machine learning[J]. Frontiers in Chemistry, 10, 864701(2022).

    [88] Shechtman Y, Weiss L E, Backer A S et al. Precise three-dimensional scan-free multiple-particle tracking over large axial ranges with tetrapod point spread functions[J]. Nano Letters, 15, 4194-4199(2015).

    [89] Kay S M[M]. Fundamentals of statistical signal processing(1993).

    [90] Li Y M, Shi W, Liu S et al. Global fitting for high-accuracy multi-channel single-molecule localization[J]. Nature Communications, 13, 3133(2022).

    [91] Li Y M, Mund M, Hoess P et al. Real-time 3D single-molecule localization using experimental point spread functions[J]. Nature Methods, 15, 367-369(2018).

    [92] Möckl L, Roy A R, Petrov P N et al. Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural network BGnet[J]. Proceedings of the National Academy of Sciences of the United States of America, 117, 60-67(2020).

    [93] Zhang Z Y, Zhang Y, Ying L et al. Machine-learning based spectral classification for spectroscopic single-molecule localization microscopy[J]. Optics Letters, 44, 5864-5867(2019).

    [94] Jeong D, Kim D. Super-resolution fluorescence microscopy-based single-molecule spectroscopy[J]. Bulletin of the Korean Chemical Society, 43, 316-327(2022).

    [95] Khater I M, Nabi I R, Hamarneh G. A review of super-resolution single-molecule localization microscopy cluster analysis and quantification methods[J]. Patterns, 1, 100038(2020).

    [96] Sage D, Pham T A, Babcock H et al. Super-resolution fight club: assessment of 2D and 3D single-molecule localization microscopy software[J]. Nature Methods, 16, 387-395(2019).

    [97] Venkataramani V, Herrmannsdörfer F, Heilemann M et al. SuReSim: simulating localization microscopy experiments from ground truth models[J]. Nature Methods, 13, 319-321(2016).

    [98] Novák T, Gajdos T, Sinkó J et al. TestSTORM: versatile simulator software for multimodal super-resolution localization fluorescence microscopy[J]. Scientific Reports, 7, 951(2017).

    [99] Lindén M, ĆGurić V, Boucharin A et al. Simulated single molecule microscopy with SMeagol[J]. Bioinformatics, 32, 2394-2395(2016).

    [100] Levet F, Julien G, Galland R et al. A tessellation-based colocalization analysis approach for single-molecule localization microscopy[J]. Nature Communications, 10, 2379(2019).

    [101] Culley S, Albrecht D, Jacobs C et al. Quantitative mapping and minimization of super-resolution optical imaging artifacts[J]. Nature Methods, 15, 263-266(2018).

    [102] Marsh R J, Pfisterer K, Bennett P et al. Artifact-free high-density localization microscopy analysis[J]. Nature Methods, 15, 689-692(2018).

    [103] Spahn C, Herrmannsdörfer F, Kuner T et al. Temporal accumulation analysis provides simplified artifact-free analysis of membrane-protein nanoclusters[J]. Nature Methods, 13, 963-964(2016).

    [104] Sieben C, Banterle N, Douglass K M et al. Multicolor single-particle reconstruction of protein complexes[J]. Nature Methods, 15, 777-780(2018).

    [105] Cardoen B, Yedder H B, Sharma A et al. ERGO: efficient recurrent graph optimized emitter density estimation in single molecule localization microscopy[J]. IEEE Transactions on Medical Imaging, 39, 1942-1956(2020).

    [106] Xu F, Ma D H, MacPherson K P et al. Three-dimensional nanoscopy of whole cells and tissues with in situ point spread function retrieval[J]. Nature Methods, 17, 531-540(2020).

    [107] Saguy A, Alalouf O, Opatovski N et al[EB/OL]. DBlink: Dynamic localization microscopy in super spatiotemporal resolution via deep learning.

    Tools

    Get Citation

    Copy Citation Text

    Tingdan Luo, Yiming Li. Deep Learning in Single-Molecule Localization Microscopy[J]. Chinese Journal of Lasers, 2022, 49(24): 2407206

    Download Citation

    EndNote(RIS)BibTexPlain Text
    Save article for my favorites
    Paper Information

    Category: Optical Diagnostics and Therapy

    Received: Aug. 8, 2022

    Accepted: Oct. 8, 2022

    Published Online: Dec. 19, 2022

    The Author Email: Li Yiming (liym2019@sustech.edu.cn)

    DOI:10.3788/CJL202249.2407206

    Topics