Advanced Photonics, Volume. 6, Issue 6, 064001(2024)

Cross-modality transformations in biological microscopy enabled by deep learning

Dana Hassan1,2, Jesús Domínguez1,3, Benjamin Midtvedt1, Henrik Klein Moberg4, Jesús Pineda1, Christoph Langhammer4, Giovanni Volpe1, Antoni Homs Corbera2, and Caroline B. Adiels1、*
Author Affiliations
  • 1University of Gothenburg, Department of Physics, Gothenburg, Sweden
  • 2CherryBiotech, Research and Development Unit, Montreuil, France
  • 3Elvesys – Microfluidics Innovation Center, Elvesys, Paris, France
  • 4Chalmers University of Technology, Department of Physics, Gothenburg, Sweden
  • show less
    References(190)

    [1] D. Murphy, M. Davidson. Fundamentals of Light Microscopy and Electronic Imaging(2012).

    [2] K. Suvarna, C. Layton, J. Bancroft. Bancroft’s Theory and Practice of Histological Techniques E-Book(2012).

    [4] I. Johnson. Molecular probes handbook: a guide to fluorescent probes and labeling technologies(2010).

    [6] S. Shashkova, M. Leake. Single-molecule fluorescence microscopy review: shedding new light on old problems. Biosci. Rep., 37, BSR20170031(2017).

    [11] J. Ferreira, L. Groc. Surface Glutamate Receptor Nanoscale Organization with Super-Resolution Microscopy (dSTORM), 35-52(2024).

    [17] X. Cao et al. Deep learning based inter-modality image registration supervised by intra-modality similarity. Lecture Notes in Computer Science, 55-63(2018).

    [19] J. Johnson, A. Alahi, L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. Lecture Notes in Computer Science, 694-711(2016).

    [20] C. Ledig et al. Photo-realistic single image super-resolution using a generative adversarial network, 15640-15649(2017).

    [21] K. Zhang et al. Negative-aware attention framework for image-text matching(2022).

    [22] T. S. Gurina, L. Simms. Histology, staining(2023).

    [25] B. Bai et al. Label-free virtual HER2 immunohistochemical staining of breast tissue using deep learning. BME Frontiers, 2022, 9786242(2022).

    [29] X. Wang et al. Single-shot isotropic differential interference contrast microscopy. Nat. Commun., 14, 2063(2023).

    [34] E. Breznik et al. Cross-modality sub-image retrieval using contrastive multimodal image representations. Sci. Rep., 14, 18798(2024).

    [35] A. Lahiani et al. Virtualization of tissue staining in digital pathology using an unsupervised deep learning approach. Lecture Notes in Comput. Sci., 47-55(2019).

    [48] Y. Hong et al. Deep learning-based virtual cytokeratin staining of gastric carcinomas to measure tumor–stroma ratio. Sci. Rep., 11, 19255(2021).

    [50] J. Y. Zhu et al. Unpaired image-to-image translation using cycle-consistent adversarial networks, 2242-2251(2017).

    [55] X. Li et al. Unsupervised content-preserving transformation for optical microscopy. Light Sci. Appl., 10, 44(2021).

    [61] R. Sanyal, D. Kar, R. Sarkar. Carcinoma type classification from high-resolution breast microscopy images using a hybrid ensemble of deep convolutional features and gradient boosting trees classifiers. IEEE/ACM Trans. Comput. Biol. Bioinf., 19, 2124-2136(2021).

    [65] Q. Dou et al. Unsupervised cross-modality domain adaptation of convnets for biomedical image segmentations with adversarial loss, 691-697(2018).

    [72] M. Dohmen et al. Similarity metrics for MR image-to-image translation(2024).

    [76] M. Luella, B. Paul, A. Javad. Generative AI in medical imaging and its application in low dose computed tomography (CT) image denoising. Applications of Generative AI, 387-401(2024).

    [77] Y. Choi et al. StarGAN: unified generative adversarial networks for multi-domain image-to-image translation, 8789-8797(2018).

    [78] J. Vasiljević et al. HistostarGAN: a unified approach to stain normalisation, stain transfer and stain invariant segmentation in renal histopathology. Knowl.-Based Syst., 277, 110780(2023).

    [82] S. Banerji, S. Mitra. Deep learning in histopathology: a review. WIREs Data Min. Knowl. Discovery, 12, e1439(2022).

    [83] J. Xu et al. Deep Learning for Histopathological Image Analysis: Towards Computerized Diagnosis on Cancers, 73-95(2017).

    [89] P. A. Moghadam et al. A morphology focused diffusion probabilistic model for synthesis of histopathology images, 1999-2008(2023).

    [90] A. Greenspan, Y. Shen, J. Keet?al.. StainDiff: transfer stain styles of histology images with denoising diffusion probabilistic models and self-ensemble. Med. Image Computing and Computer Assisted Intervention, 549-559(2023).

    [91] T. Kataria, B. Knudsen, S. Y. Elhabian. StainDiffuser: multitask dual diffusion model for virtual staining(2024).

    [92] S. Dubey, X. Xu et al. VIMS: virtual immunohistochemistry multiplex staining via text-to-stain diffusion trained on uniplex stains. Mach. Learn. Med. Imaging, 143-155(2024).

    [93] T. M. Abraham, R. Levenson. A comparison of diffusion models and CycleGANs for virtual staining of slide-free microscopy images, 1-6(2023).

    [108] C. L. Cooke et al. Physics-enhanced machine learning for virtual fluorescence microscopy, 3803-3813(2021).

    [119] E. Wolf. Progress in Optics(2008).

    [121] C. Ledig et al. Photo-realistic single image super-resolution using a generative adversarial network, 105-114(2016).

    [123] C. Chen et al. Synergistic image and feature adaptation: towards cross-modality domain adaptation for medical image segmentation, 865-872(2019).

    [124] D. Maleki, H. Tizhoosh. LILE: look in-depth before looking elsewhere: a dual attention network using transformers for cross-modal information retrieval in histopathology archives(2022).

    [126] R. Naseem et al. Cross modality guided liver image enhancement of CT using MRI, 46-51(2019).

    [127] C. Dong, D. Fleet et al. Learning a deep convolutional network for image super-resolution, 184-199(2014).

    [133] E. Kang, J. Min, J. C. Ye. A deep convolutional neural network using directional wavelets for low-dose X-ray CT reconstruction. Med. Phys., 44, e360-e375(2017).

    [134] C. Dong, J. M. Leibe, C. C. Loy, X. Tang et al. Accelerating the super-resolution convolutional neural network. Computer Vision–ECCV 2016, 391-407(2016).

    [139] M. Haris, G. Shakhnarovich, N. Ukita. Deep back-projection networks for super-resolution, 1664-1673(2018).

    [144] J.-Y. Lin, Y.-C. Chang, W. H. Hsu. Efficient and phase-aware video super-resolution for cardiac MRI, 66-76(2020).

    [145] Y. Huang, L. Shao, A. F. Frangi. Simultaneous super-resolution and cross-modality synthesis of 3D medical images using weakly-supervised joint convolutional sparse coding, 5787-5796(2017).

    [146] W.-S. Lai et al. Deep Laplacian pyramid networks for fast and accurate super-resolution, 624-632(2017).

    [163] J. Soh, S. Cho, N. Cho. Meta-transfer learning for zero-shot super-resolution, 3513-3522(2020).

    [165] H. Sahak et al. Denoising diffusion probabilistic models for robust image super-resolution in the wild(2023).

    [166] S. Gao et al. Implicit diffusion models for continuous super-resolution, 10021-10030(2023).

    [167] J. Ho et al. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23, 1-33(2021).

    [168] R. Rombach et al. High-resolution image synthesis with latent diffusion models, 10674-10685(2021).

    [169] A. Saguy et al. This microtubule does not exist: super-resolution microscopy image generation by a diffusion model, 2400672(2024).

    [170] H. Greenspan, M. Pan et al. DiffuseIR: diffusion models for isotropic reconstruction of 3D microscopic images, 323-332(2023).

    [171] S. Gao et al. GDPR Requirements for Biobanking Activities Across Europe, 10021-10030(2023).

    [172] V. Colcelli et al. GDPR requirements for biobanking activities across europe(2023).

    [173] Health insurance portability and accountability act of 1996 (HIPAA), 104-191(1996).

    [174] (2018).

    [177] The protection of personal data in health information systems–principles and processes for public health(2020).

    [180] A. S. Pillai. Utilizing deep learning in medical image analysis for enhanced diagnostic accuracy and patient care: challenges, opportunities, and ethical implications. J. Deep Learn. Genomic Data Anal., 1, 1-17(2021).

    [182] N. Forgó et al. Big data, AI and health data: between national, european, and international legal frameworks. Legal Challenges in the New Digital Age, 358-394(2023).

    [186] K. Grünberg et al. Ethical and privacy aspects of using medical image data. Cloud-Based Benchmarking of Medical Image Analysis, 33-43(2017).

    Tools

    Get Citation

    Copy Citation Text

    Dana Hassan, Jesús Domínguez, Benjamin Midtvedt, Henrik Klein Moberg, Jesús Pineda, Christoph Langhammer, Giovanni Volpe, Antoni Homs Corbera, Caroline B. Adiels, "Cross-modality transformations in biological microscopy enabled by deep learning," Adv. Photon. 6, 064001 (2024)

    Download Citation

    EndNote(RIS)BibTexPlain Text
    Save article for my favorites
    Paper Information

    Category: Reviews

    Received: Jun. 18, 2024

    Accepted: Oct. 28, 2024

    Posted: Oct. 28, 2024

    Published Online: Nov. 29, 2024

    The Author Email: Caroline B. Adiels (caroline.adiels@physics.gu.se)

    DOI:10.1117/1.AP.6.6.064001

    CSTR:32187.14.1.AP.6.6.064001

    Topics