Fig. 1. Fundamentals of structured illumination super-resolution microscopy (SIM) and the application of deep learning to the technique

Fig. 2. Schematic diagrams of SIM principle. (a) Illustration of the basic principle of SIM, showing the convolution of the original image with the illumination pattern followed by frequency decomposition to achieve expanded frequency shifting; (b) the principle of two-dimensional spectral expansion from three directions

Fig. 3. Schematic diagram of CNN convolution principle (Assuming that the convolution kernel is a 2×2 matrix, C11=A11×B11+A12×B12+A21×B21+A22×B22)

Fig. 4. Schematic diagrams of optical path correction for SIM system using CNN^[56]. (a) Optical path of the structured light super-resolution AO system and additional aberration correction process based on the CNN model, where L1‒L6 represent lenses, SLM is the spatial light modulator used for phase modulation of spatial light, DMD is the digital micromirror device used for intensity modulation, Obj1 and Obj2 are objective lenses, sCMOS is the complementary metal-oxide-semiconductor image sensor. (b) Structured illumination image with aberrations. (c) Structured illumination image after AO correction based on the CNN model. (d) Intensity distribution along the white dashed lines in (b) and (c). (e) Series of phase distributions, where (e1) represents actual aberrations, (e2) denotes algorithm-predicted aberrations, and (e3) shows phase residuals

Fig. 5. Network architecture of U-Net^[57]. U-Net consists of an encoder, a decoder, and skip connections. The encoder extracts features, the decoder restores the original image size, and the skip connections help preserve details and contextual information to achieve high-quality image segmentation

Fig. 6. Deconvolution principle diagram. After expanding the image boundaries, a 2×2 matrix is used as the convolution kernel to perform convolution operations and expand the image output size

Fig. 7. Using U-Net for super-resolution imaging^[63]. (a) Each column from left to right represents the averaged projection of the original SIM data image, the reconstruction result of the traditional SIM reconstruction algorithm, the output result of the U-Net-SIM15 network, and the output result of the U-Net-SIM3 network, respectively. Each row represents microtubules, adhesions, mitochondria, and actin, respectively. (b) The first four images represent the averaged projection of the microtubules low-intensity SIM original data image , the reconstruction result of the traditional SIM reconstruction algorithm, the output result of the U-Net-SIM15 network, and the output result of the scU-Net network. The scU-Net effectively restores individual microtubules (white triangles)

Fig. 8. Schematic diagram of GAN network, illustrating the adversarial training process between a generator and a discriminator to generate realistic data distributions

Fig. 9. Schematic diagram (left) and actual effect (right) of training CycleGAN for SIM imaging^[68]. The input is 9 low-resolution original images, and 1d_SIM (super-resolution images in one direction) and 9_SIM images (super-resolution images in three directions) are computed using traditional algorithms. Then, two generators are used to convert 1d_SIM to 9_SIM and transfer 9_SIM images to generator 1d_SIM for cycling. Finally, missing values in 1d_SIM images are filled in to reconstruct the super-resolution 3_SIM images. Image (a) is an initial image, image (b) is a 1d_SIM image generated with three original SI images in the x-direction, image (c) is a output of the network after training with 3_SIM images, image (d) is a 9_SIM image reconstructed from 9 original images as ground truth

Fig. 10. DNN-SIM network principle and imaging effects^[70]. (a) Training process of the network starts with iterative optimization of the left half of the discriminator and feedback to the generator, followed by iterative optimization of the generator on the right side and feedback to the discriminator, thus achieving alternating updates of the parameters of the generator and discriminator; (b) wide-field-of-view image; (c) deconvolved image; (d) DNN-SIM super-resolution image; (e) traditional SIM super-resolution image

Fig. 11. RED Net network architecture diagram^[71], which utilizes residual blocks and dense connections to enhance reconstruction effectiveness and image quality

Fig. 12. FLSN contains three key elements: multi-scale network, noise estimator, and bandpass attention module^[73]