Journal of Innovative Optical Health Sciences, Volume. 16, Issue 3, 2230016(2023)

Deep-learning-based methods for super-resolution fluorescence microscopy

Jianhui Liao1, Junle Qu1, Yongqi Hao2、*, and Jia Li1、**
Author Affiliations
  • 1Shenzhen Key Laboratory of Photonics and Biophotonics, Key Laboratory of Optoelectronic Devices and Systems of Ministry of Education and Guangdong Province, College of Physics and Optoelectronic Engineering, Shenzhen University, Shenzhen 518060, P. R. China
  • 2NARI Group Corporation (State Grid Electric Power Research Institute), NARI Technology Co., Ltd., Nanjing 211106, P. R. China
  • show less
    References(64)

    [1] Beiträge zur theorie des mikroskops und der mikroskopischen wahrnehmung. Arch. Mikrosk. Anat., 9, 413-418(1873).

    [2] Quantitative evaluation of software package for single-molecule localization microscopy. Nat. Methods, 12, 717-724(2015).

    [3] Breaking the diffraction resolution limit by stimulated emission: Stimulated-emission-depletion fluorescence microscopy. Opt. Lett., 19, 780-782(1994).

    [4] STED super-resolved microscopy. Nat. Methods, 15, 173-182(2018).

    [5] Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods, 3, 793-796(2006).

    [6] Imaging intracellular fluorescent proteins at nanometer resolution. Science, 313, 1642-1645(2006).

    [7] Three-dimensional localization of single molecules for super resolution imaging and single-particle tracking. Chem. Rev., 117, 7244-7275(2017).

    [8] Image reconstruction with a deep convolutional neural network in high-density super-resolution microscopy. Opt. Express, 28, 15432-15446(2020).

    [9] Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. J. Microsc., 198, 82-87(2000).

    [10] Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics. Science, 349, aab3500(2015).

    [11] Deep learning enables structured illumination microscopy with low light levels and enhanced speed. Nat. Commun., 11, 1934(2020).

    [12] Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods, 16, 1215-1225(2019).

    [13] Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Methods, 16, 103-110(2019).

    [14] Efficient image reconstruction of high-density molecules with augmented Lagrangian method in super-resolution microscopy. Opt. Express, 26, 24329-24342(2018).

    [15] Deep learning enables structured illumination microscopy with low light levels and enhanced speed. Nat. Commun., 11, 1934(2020).

    [16] ImageNet classification with deep convolutional neural networks. Proc. 25th Int. Conf. Neural Information Processing Systems (NeurIPS), F. Pereira, C. J. C. Burges, L. Bottou, K. Q. Weinberger, 1097-1105(2012).

    [17] Fully convolutional networks for semantic segmentation. Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 3431-3440(2015).

    [18] U-net: Convolutional networks for biomedical image segmentation. Int. Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI), N. Navab, J. Hornegger, W. M. Wells, A. F. Frangi, 234-241(2015).

    [19] Deep convolutional neural network for inverse problems in imaging. IEEE Trans. Image Process., 26, 4509-4522(2017).

    [20] Deep learning microscopy. Optica, 4, 1437-1443(2017).

    [21] [21] K. Simonyan, A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” preprint, https://arxiv.org/abs/1409.1556 (2015).

    [22] Deep learning massively accelerates super-resolution localization microscopy. Nat. Biotechnol., 36, 460-468(2018).

    [23] Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell, 172, 1122-1131(2018).

    [24] Content-aware image restoration: Pushing the limits of fluorescence microscopy. Nat. Methods, 15, 1090-1097(2018).

    [25] . Deep Learning: A Practitioner’s Approach(2017).

    [26] Deep learning. Nature, 521, 436-444(2015).

    [27] Deep learning a boon for biophotonics. J. Biophotonics, 13, e201960186(2020).

    [28] Deep sparse rectifier neural networks. 14th Int. Conf. Artificial Intelligence and Statistics (AISTATS), G. J. Gordon, D. B. Dunson, M. Dudík, 315-323(2011).

    [29] Interpretation of intelligence in CNN-pooling processes: a methodological survey. Neural Comput. Appl., 32, 879-898(2020).

    [30] . Deep Learning(2016).

    [31] Large-scale machine learning with stochastic gradient descent. 19th Int. Conf. Computational Statistics (COMPSTAT), Y. Lechevallier, G. Saporta, 177-186(2010).

    [32] [32] D. P. Kingma, J. Ba, “Adam: A method for stochastic optimization,” preprint, https://arxiv.org/abs/1412.6980 (2014).

    [33] [33] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, “Generative adversarial networks,” preprint, https://arxiv.org/abs/1406.2661 (2014).

    [34] Unrolled generative adversarial networks. 5th Int. Conf. Learning Representations (ICLR), 1-25(2017).

    [35] [35] M. Arjovsky, S. Chintala, L. Bottou, “Wasserstein GAN,” Preprint at https://arxiv.org/abs/1701.07875v1 (2017).

    [36] VEEGAN: reducing mode collapse in GANs using implicit variational learning. 31st Conf. Neural Information Processing Systems (NeurIPS), U. V. Luxburg, I. Guyon, S. Bengio, H. Wallach, R. Fergus, 3310-3320(2017).

    [37] Mode seeking generative adversarial networks for diverse image synthesis. 2019 IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 1429-1437(2019).

    [38] [38] D. Berthelot, T. Schumm, L. Metz, “BEGAN: Boundary equilibrium generative adversarial networks,” preprint, https://arxiv.org/abs/1703.10717 (2017).

    [39] Image-to-image translation with conditional adversarial networks. 2017 IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 5967-5976(2017).

    [40] [40] O. Ronneberger, P. Fischer, T. Brox, “U-net: convolutional networks for biomedical image segmentation,” preprint, https://arxiv.org/abs/1505.04597 (2015).

    [41] A survey on transfer learning. IEEE T. Knowl. Data. En., 22, 1345-1359(2010).

    [42] Deep-STORM: Super-resolution single-molecule microscopy by deep learning. Optica, 5, 458-464(2018).

    [43] Batch normalization: accelerating deep network training by reducing internal covariate shift. Proc. 32nd Int. Conf. Machine Learning (ICML), B. Francis andB. David, 448-456(2015).

    [44] Imagej2: Imagej for the next generation of scientific image data. BMC Bioinf., 18, 529(2017).

    [45] ThunderSTORM: A comprehensive ImageJ plug-in for PALM and STORM data analysis and super-resolution imaging. Bioinformatics, 30, 2389-2390(2014).

    [46] Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15, 1929-1958(2014).

    [47] Multiscale structural similarity for image quality assessment. 37th Asilomar Conf. Signals, Systems & Computers, 1398-1402(2003).

    [48] Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat. Methods, 18, 194-202(2021).

    [49] [49] N. Boyd, E. Jonas, H. P. Babcock, B. Recht, “DeepLoco: Fast 3D localization microscopy using neural networks,” preprint, https://www.biorxiv.org/content/10.1101/267096v1 (2018).

    [50] Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning. Nat. Methods, 16, 1323-1331(2019).

    [51] DeepSTORM3D: Dense 3D localization microscopy and PSF design by deep learning. Nat. Methods, 17, 734-740(2020).

    [52] Three-dimensional localization of single molecules for super-resolution imaging and single-particle tracking. Chem. Rev., 117, 7244-7275(2017).

    [53] Least squares generative adversarial networks. 2017 IEEE Int. Conf. Computer Vision (ICCV), 2813-2821(2017).

    [54] FALCON: Fast and unbiased reconstruction of high-density super-resolution microscopy data. Sci. Rep., 4, 4577(2014).

    [55] Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy. Nat. Biotechnol., 36, 451-459(2018).

    [56] Analyzing single molecule localization microscopy data using cubic splines. Sci. Rep., 7, 552(2017).

    [57] Multicolour localization microscopy by point-spread-function engineering. Nat. Photonics, 10, 590(2016).

    [58] Toward real-world single image super-resolution: A new benchmark and a new model. 2019 IEEE Int. Conf. Computer Vision (ICCV), 3086-3095(2019).

    [59] Model-agnostic meta-learning for fast adaptation of deep networks. 34th Int. Conf. Machine Learning (ICML),D. Precup, Y. W. Teh, 1126-1135(2017).

    [60] [60] W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-C. F. Wang, J.-B. Huang, “A closer look at few-shot classification,” preprint, https://arxiv.org/abs/1904.04232v2 (2020).

    [61] Visualizing data using t-SNE. J. Mach. Learn. Res., 9, 2579-2605(2008).

    [62] [62] K. Simonyan, A. Vedaldi, A. Zisserman, “Deep inside convolutional networks: visualising image classification models and saliency maps,” preprint, https://arxiv.org/abs/1312.6034 (2013).

    [63] Visualizing and understanding convolutional networks. 13rd European Conf. Computer Vision (ECCV), D. Fleet, T. Pajdla, B. Schiele, T. Tuytelaars, 818-833(2014).

    [64] Simple and scalable predictive uncertainty estimation using deep ensembles. 31st Int. Conf. Neural Information Processing Systems,U. von Luxburg,I. Guyon,S. Bengio,H. Wallach,R. Fergus, 6402-6413(2017).

    Tools

    Get Citation

    Copy Citation Text

    Jianhui Liao, Junle Qu, Yongqi Hao, Jia Li. Deep-learning-based methods for super-resolution fluorescence microscopy[J]. Journal of Innovative Optical Health Sciences, 2023, 16(3): 2230016

    Download Citation

    EndNote(RIS)BibTexPlain Text
    Save article for my favorites
    Paper Information

    Category: Research Articles

    Received: Sep. 6, 2022

    Accepted: Oct. 30, 2022

    Published Online: May. 25, 2023

    The Author Email: Hao Yongqi (haoyongqi@sgepri.sgcc.com.cn), Li Jia (jli@szu.edu.cn)

    DOI:10.1142/S1793545822300166

    Topics