Advanced Photonics Nexus, Volume. 2, Issue 5, 054001(2023)
Recent advances in deep-learning-enhanced photoacoustic imaging Author Presentation
Figures & Tables(18)
Fig. 1. Representations of seven major challenges in PAI, and DL-related methods to overcome them. DAS, delay-and-sum; DL, deep learning; BF-H&E, bright-field hematoxylin and eosin staining. The images are adapted with permission from Ref. 52, © 2021 Wiley-VCH GmbH; Ref. 53, © 2020 Optica; Ref. 54, © 2022 Optica; Ref. 55, © 2020 Elsevier GmbH; Ref. 56, CC-BY; Ref. 57, © 2021 Elsevier GmbH; and Ref. 58, © 2021 Elsevier GmbH.
Fig. 2. The concept of (a) a biological neural network and (b) an ANN derived from (a). (c) Schematics of a simple neural network and a DNN.
Fig. 3. Three typical neural network architectures for biomedical imaging. (a) CNN, (b) U-Net, and (c) GAN.
Fig. 4. Representative studies using DL methods to overcome limited-detection capabilities. (a) A DNN with five fully connected layers enhances bandwidth. (b) LV-GAN for addressing the limited-view problem. (c) A Y-Net generates the PA images by optimizing both raw data and reconstructed images from the traditional method. (d) A 3D progressive U-Net (3D-pUnet) to diminish the effects of limited-view artifacts and sparsity arising from cluster view detection. The images are adapted with permission from Ref. 76, © 2017 SPIE; Ref. 77, © 2020 Wiley-VCH GmbH; Ref. 78, © 2020 Elsevier GmbH; and Ref. 52, © 2021 Wiley-VCH GmbH. BW, bandwidth; DNN, deep neural network; DAS, delay-and-sum; cluster, cluster view detection; full, full view detection.
Fig. 5. Representative DL approaches compensate for low laser dosage. (a) An MWCNN that generates high-quality PA images from low-fluence PA images. (b) An HD-UNet that enhances the image quality in a pulsed-laser diode PACT system. (c) An MT-RDN that performs image denoising, superresolution, and vascular enhancement. The images are adapted with permission from Ref. 53, © 2020 Optica; Ref. 109, © 2022 SPIE; and Ref. 110, © 2020 Wiley-VCH GmbH. MWCNN, multi-level wavelet-convolutional neural network; HD-UNet, hybrid dense U-Net; MT-RDN, multitask residual dense network.
Fig. 6. Representative studies to improve the accuracy of quantitative PAI by DL. (a) Convolutional encoder–decoder type network with skip connections (EDS) to produce accurate estimates of
Fig. 7. Representative studies to optimize conventional reconstruction algorithms or replace them with DL. (a) Pixel-wise interpolation approach followed by an FD-UNet for limited-view and sparse PAT image reconstruction. (b) End-to-end U-Net with residual blocks to reconstruct PA images. (c) Two-step PA image reconstruction process with FPnet and U-Net. The images are adapted with permission from Ref. 137, © 2020 Nature Publishing Group; Ref. 138, © 2020 Optica; and Ref. 55, © 2020 Elsevier GmbH.
Fig. 8. Representative DL studies to correct the SoS and improve the accuracy of image classification and segmentation. (a) Hybrid DNN model including U-Net and Segnet to mitigate SOS aberration in heterogeneous tissue. (b) Sparse-UNet (S-UNet) for automatic vascular segmentation in MSOT images. The images are adapted with permission from Ref. 153, CC-BY; Ref. 154, © Elsevier GmbH.
Fig. 9. Representative studies using DL to solve specific issues. (a) GAN-based framework (Wasserstein GAN) to enhance the spatial resolution of AR-PAM. (b) GAN with U-Net to reconstruct superresolution images from raw image frames. (c) Deep-PAM generates virtually stained histological images for both thin sections and thick fresh tissue specimens. The images are adapted with permission from Ref. 178, © 2021 Elsevier GmbH; Ref. 180, © 2022 Springer Nature; and Ref. 58, © 2021 Elsevier GmbH. BF-H&E, brightfield hematoxylin and eosin staining; DNN, deep neural network.
Table 1. Summary of challenges facing PAI.
View tableView in ArticleTable 1. Summary of challenges facing PAI.
Section Title Challenges to be solved 3.1 Overcoming limited detection capabilities Restricted bandwidth, limited detection view, sampling sparsity 3.2 Compensating for low-dosage light delivery Low SNR in the low-dosage light-delivery system 3.3 Improving the accuracy of quantitative PA imaging Inaccuracy in quantitative estimates ( , optical absorption coefficient) 3.4 Optimizing or replacing conventional reconstruction algorithms Limitations in conventional reconstruction algorithms 3.5 Addressing tissue heterogeneity Acoustic reflection and imaging artifacts led by tissue heterogeneity 3.6 Improving the accuracy of image classification and segmentation Inaccuracy and rough classification and segmentation of PA image 3.7 Overcoming other specified issues Motion artifacts, limited spatial resolution, electrical noise and interference, image misalignment, accelerating superresolution imaging, achieving digital histologic staining
Table 2. Three representative networks.
View tableView in ArticleTable 2. Three representative networks.
Network Key feature Use case CNN Performs convolution operation for feature extraction. Image enhancement Exhibits outstanding performance in feature extraction. Image classification and object detection Captures spatial information of input data efficiently. Image segmentation U-Net Comprises an encoder–decoder structure. Image enhancement Utilizes skip connections to leverage high-resolution feature maps. Demonstrates strong performance even with small data sets. Image segmentation Excels in segmentation tasks. GAN Consists of a generator network and a discriminator network. Image generation Generates data that closely resembles real input data (generator). Discriminates between generated data and real data (discriminator). Image style transfer Engages in competitive training between the generator and discriminator. Image/data augmentation Applies for generating new data.
Table 3. Summary of overcoming the limited detection capabilities with DL approaches.
View tableView in ArticleTable 3. Summary of overcoming the limited detection capabilities with DL approaches.
Author Neural network architecture Basic network Training data set (if specified, validation is excluded) Test data set Specified task Representative evaluation results Source Data amount Gutte et al. 76 FC-DNN CNN Simulation of the breast phantom 286,300 slices (from 2863 volumes) Simulation/in vitro phantom Reduce limited-bandwidth artifacts CNR (versus DAS) 0.01 → 2.54 PC 0.22 → 0.75 Deng et al. 83 U-Net and VGG U-Net In vivo mouse liver 50 Numerical simulation data/in vitro phantom/in vivo data Reduce limited-view artifacts from the circular US array SSIM (versus DAS) 0.39 → 0.91 PSNR 7.54 → 24.34 Zhang et al. 85 DuDoUnet U-Net k-Wave simulation 1500 k-Wave simulation Reduce limited-view artifacts from the linear US array SSIM (versus U-Net) 0.909 → 0.935 PSNR 19.4 → 20.8 Lu et al. 77 LV-GAN GAN k-Wave simulation of absorbers and vessels/in vitro phantom of microsphere and vessel structure 793 pairs (absorbers)/1600 pairs (vessels)/ k-Wave simulation of absorbers and vessels/in vitro phantom (microsphere and vessel structure) Reduce limited-view artifacts from the circular US array SSIM (versus DAS) 0.135 → 0.871 30 pairs (microsphere) PSNR 9.41 → 30.38 CNR 22.72 → 43.41 22 pairs (vessel structures) Lan et al. 78 Y-Net — k-Wave simulation of segmented blood vessels from DRIVE data set 4700 k-Wave simulation/in vitro phantom/in vivo human palm Reduce limited-view artifacts from the linear US array SSIM (versus DAS) 0.203 → 0.911 PSNR 17.36 → 25.54 SNR 1.74 → 9.92 Guan et al. 89 FD-UNet U-Net k-Wave simulation:/k-Wave simulation of realistic vasculature phantom from micro-CT images of mouse brain 1000 simulation/1000 (realistic vasculature) k-Wave simulation:/k-Wave simulation of realistic vasculature phantom (micro-CT images of the mouse brain) Reduce artifacts from sparse data in the circular US array SSIM (versus DAS) 0.75 → 0.87 PSNR 32.48 → 44.84 Farnia et al. 90 U-Net U-Net k-Wave simulation from the DRIVE data set 3200 k-Wave simulation from DRIVE data set/in vivo mouse brain Reduce artifacts from sparse data in the circular US array SSIM (versus DAS) 0.81 → 0.97 PSNR 29.1 → 35.3 SNR 11.8 → 14.6 EPI 0.68 → 0.90 Guo et al. 91 AS-Net Non k-Wave simulation of human fundus culi vessel/in vivo fish/in vivo mouse 3600/1744/1046 k-Wave simulation of human fundus culi vessel/in vivo fish/in vivo mouse Reduce artifacts from sparse data and speed up reconstruction from the circular US array SSIM (versus DAS) 0.113 → 0.985 PSNR 8.64 → 19.52 Lan et al. 92 Ki-GAN GAN k-Wave simulation of retinal vessels from public data set 4300 k-Wave simulation of retinal vessels from public data set Remove artifacts from sparse data from the circular US array SSIM (versus DAS) 0.215 → 0.928 PSNR 15.61 → 25.51 SNR 1.63 → 11.52 DiSpirito et al. 93 FD U-Net U-Net In vivo mouse brain 304 In vivo mouse brain Improve the image quality of undersampled PAM images SSIM (versus zero fill) 0.510 → 0.961 PSNR 16.94 → 34.04 MS-SSIM 0.585 → 0.990 MAE 0.0701 → 0.0084 MSE 0.0027 → 0.00044 Vu et al. 95 DIP CNN In vivo blood vessels — In vivo blood vessels/non-vascular data Improve the image quality of undersampled PAM images SSIM (versus bilinear) 0.851 → 0.928 PSNR 25.6 → 31.0 Godefroy et al. 96 U-Net/Bayesian NN U-Net Pairs of PAI and photographs of leaves/Corresponded numerical simulation 500 PAI and photographs of leaves/numerical simulation Reduce limited-view and limited-bandwidth artifacts from the linear US array NCC (versus DAS) 0.31 → 0.89 SSIM 0.29 → 0.87 Vu et al. 98 WGAN-GP GAN k-Wave simulation: disk phantom and TPM vascular data 4000 (disk)/7200 (vascular) k-Wave simulation: disk phantom and TPM vascular data/tube phantom/in vivo mouse skin Reduce limited-view and limited-bandwidth artifacts from the linear US array SSIM (versus U-Net) 0.62 → 0.65 PSNR 25.7 → 26.5 Zhang et al. 100 RADL-net CNN k-Wave simulation 161,000 (including augmentation and cropping from 126 vascular images) k-Wave simulation/vascular structure phantom/in vivo mouse brain Reduce limited-view and sparsity artifacts from the ring-shaped US array SSIM (versus DAS) 0.11 → 0.93 PSNR 17.5 → 23.3 Davoudi et al. 101 U-Net U-Net Simulation: planar parabolic absorber and mouse/in vitro circular phantom/in vitro vessel-structure phantom/in vivo mouse Not mentioned/28/33/420 Simulation: planar parabolic absorber and mouse/in vitro circular phantom/in vitro vessel-structure phantom/in vivo mouse Reduce limited-view and sparsity artifacts from the circular US array SSIM (versus input) 0.281 → 0.845 Davoudi et al. 102 U-Net U-Net In vivo human finger from seven healthy volunteers 4109 (including validation) In vivo human finger Reduce the limited-view and sparsity artifacts from the US circular array SSIM (versus U-Net) 0.845 → 0.944 PSNR 14.3 → 19.0 MSE 0.04 → 0.014 NRMSE 0.818 → 0.355 Awasthi et al. 103 Hybrid end-to-end U-Net U-Net k-Wave simulation from breast sinogram images 1000 k-Wave simulation of the numerical phantom, blood vessel, and breast/ horsehair phantoms/ in vivo rat brain Super-resolution, denoising, and bandwidth enhancement of the PA signal from the circular US array PC (versus DAS) 0.307 → 0.730 SSIM 0.272 → 0.703 RMSE 0.107 → 0.0617 Schwab et al. 105 DALnet CNN Numerical simulation of 200 projection images from 3D lung blood vessel data 3000 (after cropping) Numerical simulation/in vivo human finger Reduce limited-view, sparsity and limited bandwidth artifacts SSIM (versus input) 0.305 → 0.726 Correlation 0.382 → 0.933 Choi et al. 52 3D-pUnet U-Net In vivo rat 1089 In vivo rat/in vivo mouse/in vivo human Reduce limited-view and sparsity artifacts MS-SSIM (versus input) 0.83 → 0.94 PSNR 32.0 → 34.8 RMSE 0.025 → 0.019
Table 4. Summary of studies on compensating for low-dosage light delivery.
View tableView in ArticleTable 4. Summary of studies on compensating for low-dosage light delivery.
Author Neural network architecture Basic network Training data set (if specified, validation is excluded) Test data set Specified task Representative evaluation results Source Data amount Hariri et al. 53 MWCNN U-Net Agarose hydrogel phantom: LED-based PA image and Nd:YAG-based PA image 229 Agarose hydrogel phantom of LED-based PA image/in vivo mouse Denoise PA images from low-dosage system SSIM (versus input) 0.63 → 0.93 PSNR 15.58 → 53.88 Singh et al. 111 U-Net U-Net LED-based and Nd:YAG-based tube phantom 150 LED-based phantom using ICG and MB Reduce the frame averaging SNR 14 → 20 Anas et al. 112 — CNN In vitro phantom 4536 In vivo fingers Improve the quality of PA images SSIM (versus average) 0.654 → 0.885 PSNR 28.3 → 36.0 Anas et al. 114 — CNN and LSTM In vitro wire phantom/in vitro nanoparticle phantom 352,000 In vitro phantom/in vivo human fingers Improve the quality of PA images SSIM (versus input) 0.86→0.96 PSNR 32.3 → 37.8 Rajendran et al. 109 HD-Unet U-Net k-Wave simulation 450 In vitro phantom/in vivo rat Improve the frame rate SSIM (versus U-Net) 0.92 → 0.98 PSNR 28.6 → 32.9 MAE 0.025 → 0.017 Zhao et al. 110 MT-RDN — In vivo mouse brain and ear 6696 In vivo mouse brain and ear Improve the quality from low dosage laser and downsampled data SSIM (versus input) 0.64 → 0.79 PSNR 21.9 → 25.6
Table 5. Summary of studies to improve the accuracy of quantitative PAI.
View tableView in ArticleTable 5. Summary of studies to improve the accuracy of quantitative PAI.
Author Neural network architecture Basic network Training data set (if specified, validation is excluded) Test data set Specified task Representative evaluation results Source Data amount Cai et al. 119 ResU-net U-Net Numerical simulation 2048 Numerical simulation Extract information from multispectral PA images Relative errors (versus linear unmixing) 36.9% → 0.76% Chang. et al. 120 DR2U-net U-Net Monte Carlo simulation of simulated tissue structure 2560 Monte Carlo simulation of simulated tissue structure Extract fluence distribution from optical absorption images Relative Errors (versus linear unmixing) 48.76% → 1.27% Luke et al. 121 O-Net: U-Net Monte Carlo simulation of epidermis, dermis, and breast tissue 1600 pairs (one pair has two-wavelength PA data) Monte Carlo simulation of epidermis, dermis, and breast tissue Estimate the oxygen saturation and segment Relative errors (versus linear unmixing) 43.7% → 5.15% Yang et al. 122 EDA-net — Monte Carlo and k-Wave simulation from female breast phantom 4888 Monte Carlo and k-Wave simulation based on clinically obtained female breast phantom Extract the information from the multi-wavelength PA images Relative errors (versus linear unmixing) 41.32% → 4.78% Gröhl et al. 123 Nine-layer fully connected NN CNN Monte Carlo simulation of in silico vessel phantoms 776 In vivo porcine brain and human forearm Obtain quantitative estimates for blood oxygenation No statistical results Bench et al. 124 EDS U-Net k-Wave simulation of human lung from lung CT scans/k-Wave simulation of three-layer skin model — k-Wave simulation Produce 3D maps of vascular and vessel positions Mean difference (versus linear unmixing) 6.6% → 0.3% Chen et al. 126 U-Net U-Net Monte Carlo simulation 2880 In vitro phantom Recover the optical absorption coefficient Relative error less than 10% Gröhl et.al 127 U-Net U-Net Monte Carlo and k-Wave simulations of in silico tissue 3600 Monte Carlo and k-Wave simulation of in silico Improve optical absorption coefficient estimation Estimation error (versus linear unmixing) 58.3% → 3.1% Li et al. 54 Two GANs: SEED-Net and QOAT-Net GAN Numerical simulation of phantom, mouse, and human brain/experimental data of phantom, ex vivo, and in vivo mouse 3040, 2560, and 2560/2916, 3200, and 3800 Ex vivo porcine tissue, mouse liver, and kidney/In vivo mouse Improve optical absorption coefficient estimation Relative errors (versus linear unmixing) 8.00% → 4.82% Relative errors 8.00% → 4.82% Zou et al. 125 US-Unet U-Net Monte Carlo and k-Wave simulation/in vitro phantom 2000/480 In vitro blood tube/in vivo clinical data set Improve optical absorption coefficient estimation Accuracy (versus linear unmixing) 0.71 → 0.89 Madasamy et al. 132 Network comparing: U-Net, FD U-Net, Y-Net, FD Y-Net, Deep ResU-Net, and GAN — 2D numerical simulation of retinal fundus (from Kaggle and RFMID)/3D numerical simulation of breast phantom 1858 (before augmentation)/5 3D volumes (12,288 slices after augmentation) 2D numerical blood vessel/3D numerical breast phantom Fluence correction PSNR (versus linear unmixing) 37.9 → 45.8 SSIM 0.80 → 0.96 Durairaj et al. 135 Two networks: initialization network and unmixing network — NIRFAST and k-Wave simulation Not mentioned NIRFAST and k-Wave simulation Unmix the spectral information Regardless of prior spectral information Olefir et al. 136 DL-eMSOT: bi-directional RNN with two LSTMs — Monte Carlo simulation 10,944 In vitro phantom/in vivo mouse Replace inverse problem of eMSOT Mean error (versus eMSOT) 4.9% → 1.4% Median error 3.5% → 0.9% Standard deviation 4.8% → 1.5%
Table 6. Summary of methods to optimize or replace conventional reconstruction algorithms.
View tableView in ArticleTable 6. Summary of methods to optimize or replace conventional reconstruction algorithms.
Author Neural network architecture Basic network Training data set (if specified, validation is excluded) Test data set Specified task Representative evaluation results Source Data amount Guan et al. 137 Pixel-DL U-Net k-Wave simulation: circles, Shepp-Logan, and vasculature, vasculature phantom from micro-CT images of mouse brain 1000 (circles),/1000 (realistic vasculature) k-Wave simulation: circles, Shepp-Logan, and vasculature, vasculature phantom from micro-CT images of mouse brain Reconstruct PA images from PA signal PSNR (versus TR) 17.49 → 24.57 SSIM 0.52 → 079 Waibel et al. 139 U-Net U-Net Monte Carlo and k-Wave simulation 2304 Monte Carlo and k-Wave simulation Reconstruct PA images from PA signal IQR (versus DAS) 98% → 10% Antholzer et al. 140 U-Net U-Net Numerical simulation of ring-shaped phantoms 1000 Numerical simulation of ring-shaped phantoms Reconstruct PA images from PA signal MSE (versus general CNN) 0.33 → 0.026 Lan et al. 141 DU-Net U-Net k-Wave simulation: disc phantom and segmented fundus oculi/vessels CT 4000 k-Wave simulation: disc phantom and segmented fundus oculi/ vessels CT Reconstruct PA images from PA signal PSNR (versus DAS) 26.843 → 44.47 SSIM 0.394 → 0.994 Feng et al. 138 Res-UNet U-Net k-Wave simulation: disc bread, spider (from “quick draw”), simple wires, logos, natural phantom 58,126 (80% of 27,000, 13,000, 10,800, 6000, 240, 15,000) k-Wave simulation: disc, PAT, vessel/in vitro phantom Reconstruct PA images from PA signal Vessel phantom PC (versus MRR) 0.41 → 0.80 PSNR 6.57 → 13.29 Tong et al. 55 FPnet+ U-Net U-Net Numerical simulation: brain from MRI, abdomen from MRI, vessel from DRIVE data set/in vivo mouse brain and abdomen 15,757: 2211 (brain), 8273 (abdomen), 4000 (vessel)/698 (mouse brain), 575 (mouse abdomen) Numerical simulation: brain, abdomen and liver cancer from MRI, vessel/ in vivo mouse brain and abdomen Reconstruct PA images from PA signal MSOT-Abdomen PSNR (versus FBP) 16.0532 → 30.3972 SSIM 0.2647 → 0.9073 RMSE 0.4771 → 0.0910 Yang et al. 142 RIM — k-Wave simulation of segmented blood vessels from DRIVE data set 2400 k-Wave simulation of segmented blood vessels from DRIVE Reconstruct PA images from PA signal PSNR (versus DGD) 42.37 → 44.26 Kim et al. 143 upgUNET U-Net Monte Carlo simulation 128,000 (after augmentation) Monte Carlo simulation/ in vitro metal-wire phantom/ in vivo human finger Reconstruct PA images from PA signal PSNR (versus DAS) 20.97 → 27.73 SSIM 0.208 → 0.754 Hauptmann et al. 144 Updated DGD (deep gradient descent) DGD (deep gradient descent) k-Wave simulation pf human lung from 50 whole-lung CT scans 1024 (from 50 CT scans) k-Wave simulation pf human lung from 50 whole-lung CT scans/ in vivo human palm Reconstruct PA images from PA signal PSNR (versus U-Net) 40.81 → 41.40 SSIM 0.933 → 0.945 Hauptmann et al. 145 FF-PAT U-Net k-Wave simulation of human lung from lung CT scans 1024 (from 50 CT scans) k-Wave simulation of human lung/in vivo data Reconstruct PA images from PA signal PSNR (versus BP) 33.5672 → 42.1749
Table 7. Summary of methods for addressing tissue heterogeneity.
View tableView in ArticleTable 7. Summary of methods for addressing tissue heterogeneity.
Author Neural network architecture Basic network Training data set (if specified, validation is excluded) Test data set Specified task Representative evaluation results Source Data amount Reiter et al. 147 CNN CNN k-Wave simulation 19,296 k-Wave simulation/in vitro vessel-mimicking target phantom Identify point source — Allman et al. 148 CNN consisting of VGG16/Fast R-CNN CNN k-Wave simulation 15,993 k-Wave simulation/in vivo data Identify and remove reflection artifacts Precision, recall, and AUC > 0.96 Allman et al. 157 CNN consisting of VGG16/fast R-CNN CNN k-Wave simulation 15,993 In vitro phantom Correct reflection artifact Accuracy (phantom) 74.36% Shan et al. 150 U-Net U-Net Numerical simulation from 3 cadaver CT 64,000 Numerical simulation from 1 cadaver CT Correct reflection artifacts PSNR (versus TR) 9 → 29 SSIM (versus TR) 0.2 → 0.9 Jeon et al. 56 SegU-net U-Net k-Wave simulation of in silico phantom 270 k-Wave simulation of in silico phantom/in vivo human forearm and foot Reduce speed-of-sound aberrations In silico phantom SSIM (versus pre-corrected) + 0.24
Table 8. Summary of methods for improving the accuracy of image classification and segmentation.
View tableView in ArticleTable 8. Summary of methods for improving the accuracy of image classification and segmentation.
Author Neural network architecture Basic network Training data set (if specified, validation is excluded) Test data set Specified task Representative evaluation results Source Data amount Zhang et al. 160 AlexNet/ GoogLeNet CNN k-Wave simulation from in vivo human breast (normal, cancer) 98 (normal)/ 75 (patient) k-Wave simulation from in vivo human breast Classify and segment images BI-RADS rating accuracy: 83% to 96% Jnawali et al. 163 Inception- Resnet-V2 — Ex vivo human thyroid (normal, benign, cancer) 73 Ex vivo human thyroid Detect cancer tissue AUCs for cancer, benign, and normal: 0.73, 0.81, and 0.88 Jnawali et al. 165 3D CNN CNN Thyroid cancer tissue 74 (thyroid)/ 74 (prostate) Thyroid and prostate cancer tissue Detect cancer tissue AUC, 0.72 Moustakidis et al. 166 SkinSeg — In vivo human About 26,190 (unclear description) In vivo human Identify skin morphology Per-class accuracy: 84.95% Nitkunanantharajah et al. 169 ResNet18 — In vivo human nailfold (SSc. normal) 990 (from 33 subjects) In vivo human nailfold Classify images Intra-class correlation: 0.902. AUC: 0.897 Sensitivity and specificity: 0.783 and 0.895 Chlis et al. 154 S-Unet U-Net In vivo human vasculature 98 pairs (one pair has 28-wavelength PA data) In vivo human vasculature Segment human vasculature Dice coeff. (versus U-Net): 0.75 → 0.86 Schellenberg et al. 170 nnU-Net U-Net/ CNN PA/US images of forearm, calf, and neck from 10 volunteers 144 PA and US image pairs 36 images for validation and 108 images from six volunteers Segment images Dice coeff. (versus FCNN): 0.66 → 0.85. Normalized surface distance: 0.61 → 0.89 Lafci et al. 171 U-Net U-Net In vivo mice brain, kidney, and liver 174 images from 12 mice (brain)/ 97 images from 13 mice (kidney)/ 108 images from 14 mice (liver) In vivo mice brain, kidney, and liver Segment hybrid PA/US image Dice coeff. 0.95 Boink et al. 172 L-PD — Retinal blood vessels from DRIVE data set 768 Retinal blood vessels from DRIVE dataset/ in vitro phantom Reconstruct and segment images PSNR (versus FBP): 34 → 42.5 Ly et al. 57 Modified U-Net U-Net In vivo human palm 800 In vivo human palm Segment blood vessels profile Global accuracy: 0.9938 (SegNet-5), 0.9920 (FCN-8) → 0.9953 Sensitivity: 0.6406 (SegNet-5), 0.6220 (FCN-8) → 0.8084
Table 9. Summary of methods for addressing other specified issues.
View tableView in ArticleTable 9. Summary of methods for addressing other specified issues.
Author Neural network architecture Basic network Training data set (if specified; validation is excluded) Test data set Specified task Representative evaluation results Source Amount Chen et al. 175 CNN CNN — — Simulation/in vivo rat brain Correct motion artifact — Zheng et al. 176 MAC-Net GAN Simulation 7680 Simulation/in vivo IVUS and IVOCT Correct motion artifact AIFD (versus pre-corrected) 0.1007 → 0.0075 Cheng et al. 178 WGAN-GP GAN In vivo mouse ear 528 In vivo mouse ear Transform AR-PAM images into OR-PAM PSNR (versus blind deconv.) 18.05 → 20.02 SSIM 0.27 → 0.61 PC 0.76 → 0.78 Zhang et al. 181 AOTDL-GAN/ MultiResU-Net GAN/U-Net Simulation/in vitro phantom/in vivo mouse 3500 Simulation/in vitro phantom/in vivo mouse Transform AR-PAM images into OR-PAM SNR (versus Deconv) 4.853 → 5.70 CNR 6.93 → 12.50 Lateral resolution → Dehner et al. 182 U-Net U-Net Simulated PA image/Pure electrical noise/Simulated white Gaussian noise 3000/2110/— Simulated PA image/Pure electrical noise/phantom/in vivo human breast/simulated white Gaussian noise Remove noise SNR of sinograms (versus pre-corrected) + 10.9 dB He et al. 183 GAN GAN Leaf phantom/in vivo mouse ear 236/149 Leaf phantom/in vivo mouse ear Remove noise SNR (versus input) 29.08 → 90.73 CNR 4.80 → 7.63 Kim et al. 185 MS-FD-U-Net GAN In vivo mouse ear 830 In vivo mouse ear Align bidirectional raster scanning SSIM (versus input) 0.993 → 0.994 PSNR 50.22 → 50.62 MSE 1.09 → 0.99 Gulenko et al. 184 Modified U-Net U-Net In vivo rat colorectum/ in vivo rabbit transurethral 700 In vivo rat colorectum/in vivo rabbit transurethral Remove noise Log(RMSE) (versus Segnet) 2.9 → 2.5 Log(SSIM) −2.2 → -0.25 Log(MAE) 2.6 → 2.0 Kim et al. 180 U-Net U-Net In vivo OR-PAM mouse ear/in vivo PACT mouse brain 3000/500 In vivo OR-PAM mouse ear/in vivo PACT mouse brain Accelerate localization process PSNR (versus input) 38.47 → 40.70 MS-SSIM 0.89 → 0.97 Boktor et al. 191 Pix2Pix GAN GAN Experiments 15,000 Experiments Perform virtual staining SSIM between H&E and UV-PAM: 0.91 Cao et al. 192 CycleGAN GAN Experiments 17,940 (UV-PAM)/26,565 (H&E) Experiments Perform virtual staining H&E versus UV-PAM Cell count: 5549 and 5423. Nuclear area ( ): 24.2 and 22.0. Internuclear dist.: 10.14 and 10.18 Kang et al. 58 CycleGAN GAN Experiments 400 (thin section)/800 (thick and fresh tissue) Experiments Perform virtual staining H&E versus UV-PAM Cell count: 289 and 283. Nuclear area ( ): 70.66 and 72.75
Tools
Get Citation
Copy Citation Text
Jinge Yang, Seongwook Choi, Jiwoong Kim, Byullee Park, Chulhong Kim, "Recent advances in deep-learning-enhanced photoacoustic imaging," Adv. Photon. Nexus 2, 054001 (2023)
Paper Information
Category: Reviews
Received: May. 30, 2023
Accepted: Jul. 5, 2023
Published Online: Sep. 25, 2023
The Author Email: Park Byullee (byullee@skku.edu), Kim Chulhong (chulhong@postech.edu)
© Copyright 2018-2021 | Chinese Laser Press.
All Rights Reserved 沪ICP备15018463号-20