Fig. 1. Development of SIM technology driven by deep learning

Fig. 2. Principle of SIM technology. (a) Image of a sample illuminated by wide-field mode and its spatial spectra; (b) image of a sample illuminated by a structured pattern and the spatial spectra shifting; (c) images of a sample illuminated by structured patterns with various orientations and phase shifts, and the spatial spectra of the reconstructed super-resolution image

Fig. 3. Phototoxicity reduction methods of SIM based on deep learning. (a) scU-Net architecture and its imaging results under extremely low light conditions^[44] , two cascaded U-Nets are utilized for denoising and super-resolution reconstruction, respectively; (b) schematic diagram of DFCAN^[47], Fourier channel attention mechanism is utilized to optimize the high-frequency information in the frequency domain; (c) schematic diagram of TDV-SIM^[48], the image artifacts are suppressed by combining the physical model with total deep variation regularization; (d) schematic diagram of PRS-SIM^[49], through the pixel rearrangement strategy, four groups of images with sub-pixel displacements in space but containing the same scene information are generated from a set of original data for self-supervised denoising and reconstruction

Fig. 4. Imaging speed improvement of SIM based on deep learning. (a) Structure of U-Net-SIM3 and its imaging result^[44], and the super-resolution information is inferred from three input images by training with paired data; (b) schematic diagram of the spectrum of composite structured illumination^[57]; (c) principle of eDL-cSIM^[58], by using composite structured light illumination, super-resolution information with frequency shifts in multiple directions is captured in a single exposure, and super-resolution reconstruction is carried out using an ensemble deep learning network; (d) principle of CECI-SIM^[59-60], it reduces the number of acquired frames by compressing multi-frame structured light illumination images; (e) schematic diagram of VDL-SIM^[67], it achieves video-level real-time reconstruction by combining lightweight network design and hardware acceleration

Fig. 5. Resolution improvement methods of SIM based on deep learning. (a) Schematic diagram of the RCAN^[69], and the residual network is trained by SIM images and the corresponding high-resolution confocal images to double the axial resolution of SIM; (b) schematic diagram of ADMM-DRE-SIM^[71], and the resolution improvement is achieved through iterative optimization of the untrained neural network based on the alternating direction multiplier method combined with the degradation model of the SIM system; (c) schematic diagram of ZS-DeconvNet-SIM^[72], two groups of noise-degraded SIM images are obtained by adding noise to the original data, and after SIM reconstruction of them, denoising and resolution enhancement are carried out step by step using the network

Fig. 6. Function extension of SIM based on deep learning. (a) Principle of adaptive SIM based on deep learning^[73], it enables real-time aberration prediction and correction via CNN; (b) schematic diagram of SC-SIM^[75], the structured light illumination images are spatially encoded, deflected by dispersion, and then compressed for acquisition; (c) workflow for machine learning-based classification of bacteria using SIM images^[76]; (d) workflow for automated classification of viruses using TIRF-SIM images^[77-78]