High-speed image reconstruction for nonlinear structured illumination microscopy

Jingxiang Zhang; Tianyu Zhao; Xiangda Fu; Manming Shu; Jiajing Yan; Mengrui Wang; Yansheng Liang; Shaowei Wang; Ming Lei

doi:10.1364/PRJ.540671

1. INTRODUCTION

Super resolution structured illumination microscopy (SR-SIM) has gained significant attention in the life sciences for imaging living cells [1] due to its high spatial resolution, high imaging speed, and low phototoxicity [2 –7]. However, the resolution of linear SIM (L-SIM) is lower compared with STED [8] and PALM [9], which limits its application in many scenarios. To further improve the spatial resolution of SIM, nonlinear saturated SIM (SSIM) [10,11] was developed and successfully applied to observe the photostable sample with a resolution of $< 50 nm$ , but SSIM suffered from the severe phototoxicity and low frame rates. An intriguing approach involves using the switching of chemical states to achieve saturation for addressing high-illumination intensity [12]. In this circumstance, reversibly photoswitchable fluorescent proteins (RSFPs) offer significant advantages for achieving super resolution at much lower light intensities [12 –15]. Hirvonenet et al. discussed the use of the photoswitchable protein Dronpa with SIM to achieve nanometer scale resolution [12]. With the development of the photoswitchable protein, Li et al. proposed a nonlinear structured illumination microscopy (NL-SIM) technique called patterned activation of photoswitchable fluorophores (PA NL-SIM) to achieve 45 nm resolution in approximately 20 to 40 frames using recently developed RSFPs, termed Skylan-NS [14,16]. In general, NL-SIM generates high-order harmonics of the sample spatial frequency using the nonlinear responses of the fluorescent probes, further extending the spatial resolution to more than three times the traditional diffraction limits.

In recent years, the application of super resolution microscopy in live cell imaging has led to increased interest in real-time reconstruction and display of SR-SIM images [7,17 –19]. These technologies provide the users with almost instantaneous feedback, similar to that of a traditional widefield microscope, resulting in significantly improved work efficiency [18,20]. NL-SIM collects more raw images to solve the higher-frequency spectral component, which increases the computational burden of image reconstruction compared to linear SIM. Meanwhile, the NL-SIM reconstruction methods are mainly based on Gustafsson’s algorithm, which involves several operations, including image decomposition and reconstruction, all of which are manipulated in the frequency domain [10,13,16]. Achieving real-time imaging in NL-SIM is a significant challenge due to the sophisticated analysis required to eliminate reconstruction artifacts [21 –23] and balance separated frequency components [6,23 –25]. Obtaining a single super resolution (SR) image typically takes tens of seconds, which poses a significant challenge to achieve real-time imaging in NL-SIM. Alternative approaches focus on reconstructing the SR image in the real space by summing up the weight images [26,27]; however, these efforts do not consider the out-of-focus signal and noise elimination and artifacts reduction.

Meanwhile, the accurate estimation of the illumination parameters, including wave vector, initial phase, and modulation depth, is critical to reconstruct high-fidelity SR images. Numerous algorithms for the illumination parameters estimation have emerged, which can be categorized as iteration-free and iterative methods. The iteration-free methods, such as PCA-SIM, image recombination transform (IRT), and non-iterative auto-correlation reconstruction (ACR), which significantly improve the estimation speed, still suffer from the lack of precision. In contrast, the iterative cross-correlation (COR) algorithm maintains high accuracy and robustness even at low SNRs by iteratively searching for the precise peak location, but at the cost of computational complexity.

In our previous work, we proposed a hybrid reconstruction method termed the joint space and frequency reconstruction (JSFR) method for linear SIM [17]. This simplified protocol decomposes the SR image into the multiplication of a sequence of patterned images with weighted coefficients, and omits the complex frequency operations in the conventional Wiener-SIM [26]. The JSFR-SIM is capable of millisecond reconstruction without compromising spatial resolution. However, the previous algorithm did not consider parameter estimation, leaving the reconstruction process incomplete.

In this paper, we present a fast full-process reconstruction strategy for NL-SIM, called JSFR-NL-SIM, which includes an efficient illumination parameter estimation algorithm and extends the JSFR concept to the nonlinear SIM regime for fast SIM reconstruction. The execution time of the illumination parameter estimation is reduced to 50 ms (10 ms per direction, including the wave vector, initial phase, and modulation depth) and 5 ms for the reconstruction of $512 \times 512$ -pixel-sized raw images. Experimental results demonstrate that the image quality obtained with this new method is equivalent to that of the conventional algorithm. The formulae for JSFR-NL-SIM have been rigorously derived (see Appendices A –E for detailed workflow comparison and mathematical derivation). The objective of this work is to introduce a technique for rapidly reconstructing NL-SIM images and to promote the widespread adoption of our rapid NL-SIM reconstruction algorithms in biomedical laboratories in the future.

2. METHODS

A. Main Reconstruction Workflow of JSFR-NL-SIM

Several NL-SIM implementations have been developed to date, including saturated SIM [10,11], NL-SIM based on fluorescent photoswitchable proteins [13,16], NL-SIM based on surface plasmons [28], and NL-SIM based on STED [29]. The generalized Wiener reconstruction workflow is used for NL-SIM [2]. In this method, the acquired raw images are transformed into the Fourier domain, and all spectrum components are extracted by solving a set of linear equations. The components that were extracted are then multiplied by OTF, shifted to their true position, and added together. Finally, Wiener deconvolution is used to enhance resolution. However, several issues arise from the above reconstruction workflow. First, NL-SIM inherits the linear SIM reconstruction algorithm, which makes it difficult to overcome the intrinsic limitations of complex computation, resulting in a very challenging task to improve the reconstruction speed. Second, parameter estimation is not discussed in detail in NL-SIM, which consumes a lot of time. Finally, the Wiener reconstruction workflow is susceptible to the out-of-focus signal [30,31] and related sidelobe artifacts [24,32]. To address these issues, we developed a full-process NL-SIM reconstruction workflow, termed JSFR-NL-SIM, which is capable of rapidly reconstructing high-fidelity SR images. The whole workflow is shown in Fig. 1. In general, the proposed JSFR-NL-SIM contains a computational routine and a preparation procedure. The computational routine contains two parts, a fast illumination parameters estimation algorithm and an NL-SIM reconstruction scheme based on joint spatial and frequency domain reconstruction. The preparation procedure aims to pre-calculate the optimization filter and the coefficient functions for the performance enhancement. Therefore, once the raw images are collected, the JSFR-NL-SIM is able to rapidly perform the parameters estimation and SR image reconstruction.

Figure 1.The workflow of JSFR-NL-SIM can be divided into two parts: preparation procedure and computational routine. During the preparation procedure, optimization functions and coefficient functions are required to calculate in advance. In the computational routines, parameters estimation is first performed and then 25 raw images are filtered with the pre-calculated attenuation function. The intermediate images are obtained by summing the results of multiplying the coefficient functions with the filtered images. The final SR image is then reconstructed by filtering the optimization functions.

Download full size

View all figures

B. DFT-Based Rapid Illumination Parameters Estimation

In the first part of the computational routine, a fast illumination parameters estimation algorithm based on discrete Fourier transform (DFT) implements a local discrete Fourier transform to replace the computationally expensive iterative cross correlation operation for directly locating the wave vector with subpixel accuracy. For the conventional COR algorithm, hundreds of peak searches and corresponding cross-correlation operations impose a heavy computational burden. To address these issues, we develop an iteration-free accelerated COR algorithm, which first locates the wave vector in the integer-level precision and then directly determines the peak location in the subpixel-level precision by utilizing discrete Fourier transform. Specifically, the cross-correlation between the zeroth-order spectrum and first-order spectrum is first executed to obtain the rough integer wave vector location $(u_{0}, v_{0})$ [Figs. 2(b) and 2(c)]. In order to reduce the contribution of the low-frequency signal of separated spectrum and automatically detect the peak, we apply the notch filter to the separated spectrum and cross-correlation map. Subsequently, following Eq. (1), a localized DFT is performed using matrix multiplication to upsample the cross-correlation map in a small neighborhood (for example, $1.5 \times 1.5$ pixels around the initial integer level pixel). Then the local DFT matrix centered on this location can be calculated [Fig. 2(c)] [33]: $F (u, v) = \exp (\frac{- i 2 π u x^{T}}{M α}) \cdot C (x, y) \cdot \exp (\frac{- i 2 π y v^{T}}{N α}),$ (1)where $C (x, y)$ is the cross-correlation map and the vector $(\frac{u}{M α}, \frac{v}{N α})$ is the span subpixel coordinate centered on the pixel $(u_{0}, v_{0})$ ; $α$ is the upsampling factor. This operation outputs an array with dimensions of $(1.5 κ, 1.5 κ)$ , where $κ$ is the upsampled factor. Actually, the final searching region is the neighborhood of the $(u_{0}, v_{0})$ with an upsampling factor $α$ by utilizing Eq. (1), which directly locates the peak of the wave vector in subpixel with minimal computation burden. For NL-SIM, predicting the illumination parameters under the five directions requires a total of approximately 50 ms.

Figure 2.Workflow of the parameter estimation of the proposed JSFR-NL-SIM. (a) Raw image stack. (b) Separated spectrum components. (c) Flowchart of the wave vector estimation using the proposed method. (d) Initial phase estimation. (e) SR image reconstructed by the full-process workflow.

Download full size

View all figures

C. Rapid NL-SIM Reconstruction in the Spatial Domain

In contrast to Wiener-NL-SIM, which reconstructs the SR image in the frequency domain, the JSFR-NL-SIM reconstructs the SR image in the spatial domain.

First, raw images are filtered with the designed Wiener-type filter [34], as follows: $D_{i}^{filtered} (x) = F^{- 1} {F [D_{i} (x)] G (k)},$ (2)where $G (k)$ represents the Wiener-type filter, $D_{i} (x)$ are the patterned raw images, and $D_{i}^{filtered} (x)$ are the filtered raw images. The general form of the NL-SIM reconstruction algorithm is deduced from the perspective of the spatial domain. All the filtered images $D_{i}^{filtered} (x)$ contribute to the intermediate image $I_{SR} (x)$ with the corresponding coefficient function $c_{i} (x)$ , as the following equation. Details can be found in Appendix B. $I_{SR} (x) = \sum_{i = 1}^{5} c_{i} (x) D_{i}^{filter} (x) .$ (3)

We assume the general phase step of NL-SIM raw data is $2 π (i - 1) / N$ , and specifically set to $2 π (i - 1) / 5$ in our case. The coefficient function is shown as follows: $[\begin{matrix} c_{1} (x) \\ c_{2} (x) \\ c_{3} (x) \\ c_{4} (x) \\ c_{5} (x) \end{matrix}] = {[\begin{matrix} 1 & 1 & 1 & 1 & 1 \\ 1 & \cos Δ φ & \cos 2 Δ φ & \cos 3 Δ φ & \cos 4 Δ φ \\ 0 & \sin Δ φ & \sin 2 Δ φ & \sin 3 Δ φ & \sin 4 Δ φ \\ 1 & \cos 2 Δ φ & \cos 4 Δ φ & \cos 6 Δ φ & \cos 8 Δ φ \\ 0 & \sin 2 Δ φ & \sin 4 Δ φ & \sin 6 Δ φ & \sin 8 Δ φ \end{matrix}]}^{- 1} [\begin{matrix} \frac{1}{I_{0}} \\ \frac{1}{m_{1} I_{0}} \cos (ω x + φ_{0}) \\ \frac{1}{m_{1} I_{0}} \sin (ω x + φ_{0}) \\ \frac{1}{m_{2} I_{0}} \cos [2 (ω x + φ_{0})] \\ \frac{1}{m_{2} I_{0}} \sin [2 (ω x + φ_{0})] \end{matrix}] .$ (4)

The coefficient functions $c_{i} (x)$ are only associated with the illumination parameters, which can be calculated using the proposed DFT-based parameter estimation algorithm and classical COR algorithm. According to the calculated coefficient functions, an intermediate image without deconvolution can be obtained. Finally, we multiply the intermediate image $I_{SR} (x)$ with the optimization functions for final deconvolution, as follows: $I_{SR - final} (x) = F^{- 1} {F [\sum_{d = 1}^{5} \sum_{p = 1}^{5} c_{i} (x) D_{i}^{filter} (x)] \cdot W (k)},$ (5)where $c_{i} (x)$ is the coefficient function and $W (k)$ is the optimization functions containing two filters $W_{1}$ and $W_{2}$ , which are designed to approach the ideal spectrum and suppress the spectrum’s peak to eliminate the out-of-focus signal, noise, and related sidelobe artifacts [6,35] (Appendix C).

We also demonstrate the connection between the reconstruction algorithm in the spatial domain and the Fourier domain. In the spatial domain, the final image can be represented as a linear combination of the intermediate images with the weighted coefficients, as follows (Appendix D): $R_{1} (x) = \frac{e^{i φ_{0}}}{2} F^{- 1} [\tilde{O} (k - ω) \cdot \tilde{H} (k)] - \frac{e^{- i φ_{0}}}{2} F^{- 1} [\tilde{O} (k + ω) \cdot \tilde{H} (k)], R_{2} (x) = \frac{e^{i φ_{0}}}{2 i} F^{- 1} [\tilde{O} (k - ω) \cdot \tilde{H} (k)] - \frac{e^{- i φ_{0}}}{2 i} F^{- 1} [\tilde{O} (k + ω) \cdot \tilde{H} (k)], R_{3} (x) = \frac{e^{2 i φ_{0}}}{2} F^{- 1} [\tilde{O} (k - 2 ω) \cdot \tilde{H} (k)] - \frac{e^{- 2 i φ_{0}}}{2} F^{- 1} [\tilde{O} (k + 2 ω) \cdot \tilde{H} (k)], R_{4} (x) = \frac{e^{2 i φ_{0}}}{2 i} F^{- 1} [\tilde{O} (k - 2 ω) \cdot \tilde{H} (k)] - \frac{e^{- 2 i φ_{0}}}{2 i} F^{- 1} [\tilde{O} (k + 2 ω) \cdot \tilde{H} (k)],$ (6)where $ω$ and $φ_{0}$ are denoted as the angular frequency and the initial phase of the illumination patterns, respectively. In the Fourier domain, the above equations can be written as $\tilde{O} (k - ω) \cdot \tilde{H} (k) = e^{- i φ_{0}} F [R_{1} (x) + i R_{2} (x)], \tilde{O} (k + ω) \cdot \tilde{H} (k) = e^{i φ_{0}} F [R_{1} (x) - i R_{2} (x)], \tilde{O} (k - 2 ω) \cdot \tilde{H} (k) = e^{- 2 i φ_{0}} F [R_{3} (x) + i R_{4} (x)], \tilde{O} (k + 2 ω) \cdot \tilde{H} (k) = e^{2 i φ_{0}} F [R_{3} (x) - i R_{4} (x)] .$ (7)

We demonstrate the inherent connection of five separated spectra in the frequency domain $O (k) H (k)$ , $O (k \pm ω) H (k)$ , and $O (k \pm 2 ω) H (k)$ with the intermediate images, and verify that the result of the reconstruction image recovered in the spatial domain is equivalent to that in the Fourier domain.

3. RESULTS

A. JSFR-NL-SIM Improved the Resolution without Reconstruction Time Increased

We first validated the reconstruction performance by applying the JSFR-based workflow in linear SIM, NL-SIM (1HOH, one higher-order harmonic), and NL-SIM (2HOH, two additional higher-order harmonics) [7]. The SR image obtained with JSFR-SIM improved the resolution by up to a factor of two and the resolution was further improved by approximately three times and four times by generating 1HOH and 2HOH with JSFR-NL-SIM [see Figs. 3(c)–3(h)]. Furthermore, we have compared the reconstruction time of the JSFR-NL-SIM with that of the HiFi-NL-SIM [15,18] in both the NL-SIM (1HOH) and NL-SIM (2HOH) scenarios [Fig. 3(i)]. The results demonstrate that for the circumstance of NL-SIM (1HOH), the execution time of JSFR-NL-SIM’s reconstruction procedure is around 5.5 ms, when handling input images with $512 \times 512$ pixels, while in the scenario of NL-SIM (2HOH), the reconstruction time is around 8.9 ms. The reconstruction speed was compared using MATLAB (R2023a, Math Works Inc., Natick, Massachusetts, United States). The reconstruction codes for HiFi-NL-SIM and JSFR-NL-SIM (both CPU and GPU versions) were executed on a personal computer (Intel Core i7-13700K@2.1 GHz, DRR4 3200 MHz 64 GB, NVIDIA GeForce RTX 4070ti 12 GB, Samsung 860 EVO 500 GB SSD) running Windows 10.

Figure 3.Simulation results to demonstrate the quality of JSFR-NL-SIM. (a) Ground truth. (b) Widefield image. (c) SR image reconstructed by JSFR-SIM. (d) SR image reconstructed by JSFR-NL-SIM. (e) SR image reconstructed by JSFR-NL-SIM with two additional higher-order harmonics (2HOH). (f) Zoom-in views of the blue boxes and yellow boxes in (b)–(e). (g) Spectra of (b)–(e). (h) Intensity profiles of green lines in (b)–(e). (i) Execution time of JSFR-based workflow and HiFi-based workflow in the same GPU environment; unit, millisecond. (j) Ideal spectrum.

Download full size

View all figures

Nonlinear structured illumination microscopy (NL-SIM) produces an infinite number of harmonics, corresponding to theoretically infinite resolution. However, resolving for more high-frequency information requires the acquisition of more raw images, which will undoubtedly increase the computational burden of image reconstruction. Specifically, detecting two higher-order harmonics requires 63 raw images to reconstruct the nearly isotropic resolution SR image, which involves more complex computations and further limits the feasibility of real-time imaging. In such scenarios, JSFR-NL-SIM has more advantages in reconstructing SR images, as demonstrated in Fig. 3. The execution time for reconstructing linear SIM, nonlinear SIM with one higher-order harmonic (1HOH), and nonlinear SIM with two additional higher-order harmonics (2HOH) with an image pixel number of $512 \times 512$ was simulated using the JSFR-NL-SIM and HiFi-NL-SIM. The simulation results indicate that the execution time of the HiFi-NL-SIM increases significantly with the number of spectra to be decomposed, while the execution time of the JSFR-NL-SIM increases linearly, with all processes running in milliseconds. This demonstrates the superiority of the JSFR-NL-SIM in terms of reconstruction time. However, due to the low SNR, the second higher-order harmonic signal is easily submerged in the noise. Therefore, when more efficient fluorophores become available in the future, we believe that JSFR-NL-SIM will provide more comprehensive support.

B. JSFR-NL-SIM Accelerates the Reconstruction Significantly

Figure 4.Performance analysis of JSFR-NL-SIM reconstructed on raw data with high out-of-focus signal and noise. (a) Widefield image of F-actin filaments. (b)–(d) SR images reconstructed using Wiener-NL-SIM provided by BioSR, HiFi-NL-SIM, and JSFR-NL-SIM. (e) Magnified images of (a)–(d), corresponding to the green box of (a), where out-of-focus signal and filament artifacts are produced by GT-SIM but eliminated in HiFi-NL-SIM and JSFR-NL-SIM. In (f) the yellow arrows indicate the sidelobe artifact while in (g), the red arrows represent the out-of-focus signal. The intensity profiles of both yellow lines are shown in (h). The scale bars are 5 μm in (a)–(d), 1 μm in (e), and 500 nm in (f) and (g).

Download full size

View all figures

For the setting of the reconstruction parameters, we set the strength and Wiener parameter in the Wiener-type filter and two constants of the optimization filters $w_{1}$ and $w_{2}$ to 0.91, 0.5, 0.5, and 0.1, respectively. The discussion of the reconstruction parameters is shown in Fig. 5. In particular, we always set the strength parameter of the Wiener-type filter to 0.9–0.99 and keep the Wiener parameters $w_{1}$ and $w_{2}$ as 0.5, 0.5, and 0.1, respectively.

Figure 5.Discussion of reconstruction parameters on final SR image. (a)–(d) The effect of attenuation parameter, Wiener-type filter’s parameter, optimization filter1 parameter, and optimization filter2 parameter. The intensity of (a)–(d) is the white line along plane (e). (f)–(h) Intensity of the images filtered with Wiener-type filter, optimization filter1, and optimization filter2. The scale bars are 5 μm in (a)–(c) and 1 μm in (d).

Download full size

View all figures

C. JSFR-NL-SIM Produces High-Fidelity SR Images

Finally, we demonstrate the ability of JSFR-NL-SIM to produce high-fidelity images with the F-actin filaments, as shown in Figs. 4 and 6. In the widefield image, the clear visualization of the intersected filaments is masked with the highly out-of-focus signal, obstructing the observation of cellular details [Fig. 6(a)]. After executing different reconstruction approaches, the out-of-focus signal has been effectively eliminated. However, the ability of GT-SIM to restrain sidelobe artifacts and honeycomb artifacts induced by a strong out-of-focus signal is moderate. By employing the HiFi-NL-SIM and JSFR-NL-SIM, these artifacts are eliminated successfully [Figs. 6(a) and 6(b)]. In addition, the SR image reconstructed by JSFR-NL-SIM maintains the linear relationship from the intensity profiles [Figs. 4(h) and 6(c)]. We demonstrate the whole image quality using decorrelation analysis [38] [Fig. 6(f)], where the resolution of the widefield image is approximately 260 nm and that of the super resolution image is about 70 nm. We also estimate the local resolution [39] by subdividing the image into smaller tiles and calculating its histogram, where the minimum resolution is 48 nm and that of maximum is 113 nm [Figs. 6(d) and 6(e)].

Figure 6.Reconstruction results from the BioSR dataset. (a) displays the reconstruction results of F-actin filaments, including widefield image and the SR images reconstructed by traditional Wiener-NL-SIM (termed GT-SIM, provided by BioSR dataset), HiFi-NL-SIM, and the proposed JSFR-NL-SIM. (b) provides zoomed-in views of (a), corresponding to the white box in (a). (c) Intensity profiles of the white lines in (b). (d), (e) Location resolution estimation and corresponding histogram of blue box in (a). (f) Decorrelation analysis to evaluate resolution. The scale bars are 5 μm in (a) and 1 μm in (b).

Download full size

View all figures

4. CONCLUSION

In conclusion, nonlinear SIM employs the nonlinear fluorescence response of fluorescent probes to generate higher-order spatial frequency information. Theoretically, if these high-frequency components can be resolved, the spatial resolution of nonlinear SIM can be improved to an infinite degree. The advent of new reversibly photoswitchable fluorescent proteins has led to the emergence of nonlinear SIM techniques that require low excitation power. Nonetheless, the reconstruction of nonlinear SIM images presents a more formidable challenge in comparison to linear SIM. First, nonlinear SIM necessitates the acquisition of 25 to 63 raw images for the reconstruction of a super resolution image, thereby imposing a significant computational burden. Second, nonlinear SIM necessitates the use of a greater number of illumination angles, and the prediction of parameters is a more time-consuming process than in linear SIM. Third, the high-frequency components of nonlinear SIM are susceptible to noise interference, and the resulting computational artifacts are typically substantial, leading to a deterioration in image fidelity. Consequently, research in this field has made limited progress over time. The proposed JSFR-NL-SIM effectively addresses the aforementioned technical challenges encountered by nonlinear SIM and has been demonstrated to be the most rapid in the SIM field. A complete nonlinear SIM image reconstruction process can be controlled in the millisecond scale. It is noteworthy that the computing speed of the proposed discrete-Fourier-transform-based parameter prediction method is considerably faster than that of all current parameter prediction techniques, and it is also applicable to linear SIM.

Our objective is to present a technique for real-time reconstruction of NL-SIM images and to promote the widespread use of NL-SIM in biomedical laboratories. JSFR-NL-SIM was extensively tested on the numerous data from open source BioSR datasets [36]. Rigorous spatial resolution criteria were also established and compared with other NL-SIM methods. The final image quality and resolution obtained by JSFR-NL-SIM are consistent with traditional NL-SIM, as verified by theoretical derivation and experimental results. JSFR-NL-SIM is readily compatible with various methods, including HiFi methods [6], Hessian denoising [4], and sparse deconvolution [5] to improve the final SR images. By integrating with the VIGOR framework proposed by Markwirth et al. [18], JSFR-NL-SIM has the potential to achieve real-time, multicolor observation with a larger field of view and reduce the latency between measurement and reconstructed image display. Additionally, by combining with the joint spectrum optimization strategy presented by Wen et al. [6,37], reconstruction artifacts in JSFR-NL-SIM can be further eliminated. Furthermore, JSFR-NL-SIM enables fast acquisition of SR images, which facilitates data acquisition. Moreover, JSFR-NL-SIM is expected to serve as the high-quality ground truth for training linear SIM in deep learning in the future [40 –43].

However, the implementation of nonlinear SIM is constrained by the complexities associated with sample preparation and the inherent limitations of fluorescent probes. Consequently, the real-time nonlinear reconstruction experiments remain incomplete. As fluorescent dyes continue to evolve, we anticipate that JSFR will facilitate the future realization of real-time NL-SIM.

Acknowledgment

Acknowledgment. We acknowledge the valuable contribution made by our former team members, Dr. Zhaojun Wang and Dr. Xing Zhou, to JSFR-SIM.

Category: Imaging Systems, Microscopy, and Displays

Received: Aug. 28, 2024

Accepted: Dec. 20, 2024

Published Online: Mar. 3, 2025

The Author Email: Ming Lei (ming.lei@mail.xjtu.edu.cn)

DOI:10.1364/PRJ.540671

Input Image Size (Pixel)	Execution Time of the DFT Algorithm (ms)	Execution Time of the COR Algorithm (ms)
$512 \times 512$	$49.87 \pm 1.5$	$111.8 \pm 4.9$
$1024 \times 1024$	$118.6 \pm 4.7$	$213.5 \pm 6.8$

Input Image Size (Pixel)	Execution Time of the DFT Algorithm (ms)	Execution Time of the PCA Algorithm (ms)	Execution Time of the COR Algorithm (ms)
$512 \times 512$	$30.04 \pm 1.3$	$33.01 \pm 1.3$	$68.67 \pm 2.0$
$1024 \times 1024$	$76.32 \pm 2.2$	$77.31 \pm 2.1$	$124.1 \pm 4.1$