1Smart Computational Imaging Laboratory (SCILab), School of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
2Smart Computational Imaging Research Institute (SCIRI) of Nanjing University of Science and Technology, Nanjing 210019, China
3Jiangsu Key Laboratory of Spectral Imaging & Intelligent Sense, Nanjing 210094, China
The transport-of-intensity equation (TIE) enables quantitative phase imaging (QPI) under partially coherent illumination by measuring the through-focus intensities combined with a linearized inverse reconstruction algorithm. However, overcoming its sensitivity to imaging settings remains a challenging problem because of the difficulty in tuning the optical parameters of the imaging system accurately and because of the instability to long-time measurements. To address these limitations, we propose and experimentally validate a solution called neural-field-assisted transport-of-intensity phase microscopy (NFTPM) by introducing a tunable defocus parameter into neural field. Without weak object approximation, NFTPM incorporates the physical prior of partially coherent image formation to constrain the neural field and learns the continuous representation of phase object without the need for training. Simulation and experimental results of HeLa cells demonstrate that NFTPM can achieve accurate, partially coherent QPI under unknown defocus distances, providing new possibilities for extending applications in live cell biology.
【AIGC One Sentence Reading】:NFTPM, a novel microscopy method, integrates neural fields with tunable defocus for accurate phase imaging under unknown defocus distances, advancing live cell imaging applications.
【AIGC Short Abstract】:Neural-field-assisted transport-of-intensity phase microscopy (NFTPM) addresses challenges in quantitative phase imaging by integrating a tunable defocus parameter into the neural field. This innovative approach eliminates the need for weak object approximation, incorporates physical priors of image formation, and learns the phase object's continuous representation without training. NFTPM enables accurate, partially coherent imaging under unknown defocus distances, expanding possibilities in live cell biology applications.
Note: This section is automatically generated by AI . The website and platform operators shall not be liable for any commercial or legal consequences arising from your use of AI generated content on this website. Please be aware of this.
1. INTRODUCTION
Quantitative phase imaging (QPI) has gained increased interest in optical microscopy research for its capability to quantify optical thickness and morphologies of unlabeled samples [1–3]. The QPI approach can be categorized into iterative and deterministic methods [4–6], where the deterministic method requires the establishment of an analytical expression for the object phase with respect to the measured intensity images. Given that the image formation process in QPI is inherently non-linear, linearization approaches are commonly invoked to facilitate solving for phase as a function of intensity measurements. For example, as a well-established deterministic phase retrieval approach, the transport-of-intensity equation (TIE) applies paraxial approximation and slowly-varied approximation to linearize the phase retrieval problem and can recover the quantitative phase by utilizing intensity images at multiple axially defocused planes [6,7]. Under partially coherent illumination, TIE is expected to achieve improved spatial resolution beyond the coherent diffraction limit [8]. Nevertheless, in a conventional microscope with circular illumination, partial coherence tends to diminish the phase contrast, resulting in compromised imaging resolution [9].
To achieve high-resolution and high-contrast QPI, the annular illumination (AI) matching objective numerical aperture (NA) has been employed in deconvolution-based TIE, referred to as AI-TIE [10]. AI-TIE strongly boosts the phase contrast and significantly improves the practical imaging resolution to a 2-fold objective NA. The strong phase contrast is ultimately transformed to the quantitative phase images by WOTF (weak object transfer function) inversion, yielding high-quality results with enhanced resolution. However, AI-TIE is usually limited to weak scattering samples since it linearizes the image formation model by invoking weak object approximation with ignoring higher-order terms in the complex transmittance of the sample. In addition, WOTF is a function directly related to the light source distribution, objective pupil function, and defocus distance. Once WOTF is determined, AI-TIE is not capable of adaptively adjusting optical parameters such as the defocus distance during the imaging process. Therefore, such TIE-based methods may result in degraded quality of phase retrieval due to the inaccurate inverse reconstruction for nonweak objects or cases where optical parameters are incorrect.
In contrast to the aforementioned physics-based approaches [6,10], data-driven deep learning methods can establish the nonlinear pseudo-inverse mapping relation between the defocused intensity and the object phase [11–14], bypassing the obstacle of “solving nonlinear ill-posed inverse problems.” Essentially, the major reason for the success of deep learning is the abundance of training data and the explicit agnosticism from a priori knowledge of how such data are generated [15]. However, high-quality paired data acquisition in experiments requires professional supervision and extensive labor. Furthermore, the lack of data diversity will restrict its generalization to out-of-domain cases with dissimilar optical parameters. Thus, the data-driven deep learning methods tend to fail in situations where it is difficult to obtain a large amount of high-quality paired data from a variety of different imaging systems.
To overcome the above limitations, researchers have developed untrained network approaches by incorporating physical priors into deep neural networks, such as the deep phase decoder [16] and PhysenNet [17]. These methods aim to achieve nonlinear optimization by minimizing the error between the prior model-generated image and the actual measurement. Their superiority lies in introducing neural networks as advanced regularization for automatic tuning. For instance, deep image prior (DIP) method can use a randomly initialized neural network as a prior to solve inverse problems such as pixel super-resolution [18]. Especially, the BlindNet method takes distance uncertainty into account and further addresses the phase retrieval problem with unknown defocus distance [19]. Additionally, we have witnessed the rise of neural field (NF), which has become a prominent self-supervised learning method [20]. NF can represent a three-dimensional (3D) scene as a continuous field, which is parameterized by a lightweight multilayer perceptron (MLP, i.e., fully connected network) and trained without ground truth data. In conjunction with computational imaging techniques, NF typically dispenses with training on a dataset and iterates the MLP network directly on the test data until the desired physical quantities are recovered, similar to physics-driven untrained network approaches. For example, NF can be incorporated into 3D diffraction tomography [21] or two-dimensional (2D) microscopy such as lensless microscopy [22] and Fourier ptychographic imaging [23]. However, these physics-driven deep learning methods only involve coherent imaging and are unsuitable for partially coherent imaging scenarios. In fact, considering partial coherence in phase retrieval helps to yield accurate results thanks to its better alignment with the actual situation [24]. Nevertheless, it needs to introduce additional parameters (such as coherence parameter) to establish a more complete forward model. Consequently, it remains a challenge to achieve stable partially coherent QPI under varying optical parameters.
In this work, we present a partially coherent QPI approach by using a neural field and taking the Abbe imaging model [25] as the physical prior. The proposed method, termed neural-field-assisted transport-of-intensity phase microscopy (NFTPM), is actually a gradient-based iterative algorithm. It drives a coordinate-based MLP through the physical prior to represent the phase distribution as a neural field and optimizes the MLP using the gradient computed by backpropagation in both the physical model and the MLP model. This framework empowers NFTPM to concurrently adjust the defocus distance of the physical model by introducing a tunable defocus parameter, enabling stable QPI under unknown defocus distance. Moreover, NFTPM is applicable to non-weak phase objects, since the weak object approximation is not applied to the forward image formation. Instead of an image-to-image 2D CNN, NFTPM forms a point-to-point mapping function from spatial coordinates to phase values, which effectively constrains the solution space and renders single-shot QPI possible. Unlike untrained networks based on coherent imaging systems, NFTPM can adapt to various partially coherent illuminations, which is validated by simulations under circular illumination and annular illumination. Furthermore, based on a bright-field microscope equipped with annular NA-matched illumination [26,27] formed by sparsely distributed light-emitting diode (LED) elements, we realize stable QPI of unstained Henrietta Lacks (HeLa) cells, demonstrating that NFTPM is a valid approach for adaptive correction of defocus aberration during the long-term phase microscopy. Given the simplicity and effectiveness of the NFTPM method, it promises to advance the integration of partially coherent imaging with physics-driven deep learning and open new possibilities for robust non-interferometric QPI in dynamic optical environments.
2. METHODS
A. Reconstruction Algorithm of NFTPM
The schematic diagram of NFTPM is outlined in Fig. 1(a), and Fig. 1(b) illustrates the image formation process in a partially coherent microscope, which corresponds to the physics prior used to drive NFTPM to perform phase recovery. The pipeline of NFTPM comprises a radial encoding module [21] and a 5-layer MLP ( is the weights) that maps 2D spatial coordinate to phase value , which can finally represent the phase as a neural field . We adopt grid coordinates for the field of view (FOV) of interest (generally ), and the coordinates correspond to pixels on the image sensor. Initially, we utilize the radial encoding module to map densely distributed two-dimensional coordinate points to sparsely distributed high-dimensional space, thus allowing the MLP to better discriminate between different coordinate positions in order to characterize high-frequency information. For , radial encoding can be expressed as where is the encoded feature, and is the transformation matrix used for frequency expansion, which can be defined as where is the number of the expanded frequencies. The purpose of introducing frequencies is to characterize features at various scales in the radial positions. contains multiple rotation matrices, and it can be specified as where , and is the number of rotation intervals. The rotation further enables the MLP to respond to features at diverse orientations, allowing for better feature representation and avoiding noise [21]. is applied to flatten concatenated matrices into a vector, which for matrices and can be defined as Here, we simplify the radial encoding module by replacing Eq. (3) with and including into the encoded feature. Hence, can be processed like the 2D Fourier series expansion where , , and . To retrieve the phase distribution, we input into the -layer MLP of the neural field (), incorporating the following processing modules. (1) Linear module , converting into hidden features with channels (). (2) Linear module for hidden features (). (3) The last linear transformation . (4) Leaky rectified linear unit (LeakyReLU) (). (5) Sigmoid activation function . Specifically, let the th feature be denoted as , and then the th feature is given by where , and . The phase value can be represented as ( is the output of the -layer MLP), and can be reshaped as an image of phase distribution. For most biological samples, the complex transmittance can be expressed as . In a typical optical imaging system, the source with distribution at the aperture diaphragm plane ( corresponds to the 2D coordinates in Fourier space) provides partially coherent illumination, resulting in an image captured at the image plane, where is the Fourier transform of , and TCC (transmission cross-coefficient) [10,28] satisfies the following relation: where represents the complex pupil function of the imaging system, is the defocus distance along the optical axis, is the wavenumber, and is a circular function determined by the objective NA and wavelength . The TCC formula is an abstraction for the spectral coupling of a light source to an objective pupil, intrinsically characterizing the imaging system compatible with partially coherent illumination. When the illumination distribution of the imaging system is specified, the captured image is determined by the sample’s inherent property (phase delay ) and the defocus distance . Therefore, we can use a function to represent the image formation model of .
Figure 1.(a) Schematic diagram of our proposed NFTPM method. (b) The physics prior (forward image formation model) of NFTPM.
In order to achieve phase retrieval with defocus distance prediction, the uncertain defocus distance can be incorporated into the computational graph as a tunable parameter to be optimized along with the MLP. The trade-off in determining the optimal solution of NFTPM is to ensure the accuracy of the predicted defocus distance while minimizing the error between the generated intensity image and the measurement. Given a captured intensity , the spatial coordinates are fed into to obtain the phase, which is then processed through the physical model to generate intensity for comparison with using the mean square error (MSE) loss function. The above operations can be abstracted into an optimization problem ⏧⏧where ⏧ is the retrieved phase, and is the predicted defocus distance. The optimization is executed based on backpropagation and the gradient descent algorithm [29], and the specific optimization process is described in Section 7 of Ref. [30]. It is worth mentioning that the samples are assumed as pure phase objects in NFTPM, so the phase contrast provided by a single-shot defocused intensity is sufficient for precise phase recovery based on the principle of deep image prior [18,31].
B. Experimental Setup
Neural-field-assisted transport-of-intensity phase microscopy can be easily implemented on a commercial inverted bright-field microscope (IX83, Olympus, Japan) assisted by programmable LED array illumination due to the advantage of non-interferometric measurements. The LED array provides quasi-monochromatic illumination with a center wavelength of 525 nm and spectral bandwidth of 20 nm. These LED elements can be controlled to turn on to form point, circle, or annulus patterns by a field-programmable gate array (FPGA) unit (EP4CE10E22C8N, Intel, US). Twelve annularly distributed LED elements were selected in the array, with the center of the circle coinciding with the optical axis to provide matched annular illumination with a maximum illumination NA of 0.4. We utilized a Bertrand lens, positioned in an eyepiece observation tube in place of the normal eyepiece, to examine the rear focal plane of the objective lens. This examination is crucial for confirming that the circular illumination is centered precisely in the field of view or that the annular illumination is accurately inscribed within the objective lens’s pupil. A CMOS camera (Hamamatsu ORCA-Flash 4.0 C13440) with a resolution of and a pixel size of 6.5 μm was used to record the intensity information under a detection objective (10×/0.4 UPLSAPO, Olympus). This study was conducted on a workstation equipped with an Intel i9-10900K 3.70 GHz CPU and an NVIDIA GeForce RTX 3090 GPU. The proposed algorithm was operated by Python 3.7.16 and PyTorch 1.12.1.
3. RESULTS
A. Comparison with TIE, AI-TIE, BlindNet, and GS Algorithms
To validate the effectiveness of NFTPM in partially coherent QPI, we conducted simulations to compare the proposed NFTPM with TIE, AI-TIE, BlindNet [19], and GS algorithms [4,5] under both coherent illumination and circular illumination with a coherence parameter (denoted by , illumination NA/objective NA) of 0.85. It is noteworthy that AI-TIE here refers to all deconvolution-based TIE methods, adapted to coherent (point), partially coherent illuminations and not limited to annular illumination case. As shown in Figs. 2(a1) and 2(b1), the phase distribution of a HeLa cell (0–1 rad) was used to simulate an intensity image at a defocus distance of 5 μm, defined within a grid of pixels (pixel size is 6.5 μm). The objective NA is 0.4 (20× magnification), and the wavelength of monochromatic illumination is 550 nm. The comprehensive comparison under coherent illumination is detailed in Figs. 2(a2)–2(a6). Except for BlindNet and NFTPM (randomly given an initial value as the defocus distance, e.g., μ), other methods were provided with the correct defocus value (μ). It can be observed that all methods achieve accurate phase recovery due to the exact match of the physical model and the optical parameters. Specifically, NFTPM and BlindNet show the ability to correctly predict the defocus distance, enabling robust QPI even with incorrect initialization of the defocus parameter. However, since BlindNet, TIE, and GS methods ignore the effect of partial coherence on the forward image formation process, the physical priors used by these methods do not accurately apply to the circular illumination situation, leading to a significant loss of high-frequency information in the partially coherent QPI [Figs. 2(b2)–2(b4)].
Figure 2.Comparison of phase (0–1 rad) reconstruction results of a simulated cell sample using NFTPM, BlindNet, TIE, AI-TIE, and GS methods under different illumination settings and a defocus distance of 5 μm. (a) Results under coherent illumination. (b) Results under partially coherent illumination (). (c) Comparison between AI-TIE and NFTPM for the large phase (0–6 rad) under partially coherent illumination (). (d) Comparison between BlindNet and NFTPM after downsizing the network under coherent illumination. (e), (f) Phase line profiles corresponding to (c), (d).
In contrast, AI-TIE and NFTPM demonstrate better performance at [Figs. 2(b5) and 2(b6)], as they establish a nonlinear forward model that conforms to partially coherent illumination by considering illumination distribution in modeling. Besides, we extended the simulated phase range to 0–6 rad (non-weak object) to validate that NFTPM is beyond weak object approximation. The results in Fig. 2(c) show that AI-TIE suffers from low-frequency underestimation, and Fig. 2(e) quantitatively reflects the inaccuracy of the phase image recovered by AI-TIE, while the result of NFTPM is consistent with the ground truth. It is worth noting that NFTPM, with a parameter count of (MLP) and the inference time of s, outperforms BlindNet in terms of speed, which requires a much larger parameter count of (UNet) and a comparatively longer inference time of s. Furthermore, when the network size is reduced (fewer channels per layer), the representational capacity of 2D UNet () is significantly weaker, resulting in the deterioration of phase retrieval in BlindNet [Fig. 2(d)] and the mismatch profile [Fig. 2(f)], while NFTPM () remains robust. In addition, as shown in Fig. S1 in Ref. [30], we also discuss the impact of hyperparameter tuning () on NFTPM, elucidating the stability of NFTPM against the layer changes in the MLP as well as the significance of radial encoding for the high-frequency characterization (see Section 1 in Ref. [30] for detailed analysis).
B. Verification of QPI at Unknown Defocus Distances for Different Illuminations
Further simulations are shown to verify that NFTPM can accurately recover the phase without prior knowledge of the defocus distance in different partially coherent illuminations. In Figs. 3(a1)–3(a4), we simulated intensity images at μ under circular illuminations (, 0.40, and 0.75) and annular NA-matched illumination. We randomly provided an incorrect initial value of the defocus distance (10 μm) for the AI-TIE and NFTPM (the robustness of NFTPM to defocus distance initialization is verified in Fig. S2 and Section 2 of Ref. [30]) and evaluated the quality of the retrieved phase using the root mean square error (RMSE). In Figs. 3(b1)–3(b4), severe artifacts appear in the results of AI-TIE, as the phase transfer function (PTF) is mis-estimated due to the uncorrected value. In contrast, NFTPM achieves high-precision phase retrieval () in diverse illuminations and accurately predicts the defocus value based on the tunable defocus parameter . We also supplement simulations under other special illuminations (e.g., asymmetric semicircular illumination) to display the adaptability of NFTPM to arbitrary source distribution (see Fig. S3 and Section 3 of Ref. [30] for details). As depicted in Figs. 3(c1)–3(c4), the RMSE of the phase recovered by NFTPM progressively decreases with the increasing maximum illumination angle, demonstrating that NFTPM has higher imaging accuracy at large illumination angles in the absence of noise. However, under noisy conditions (Gaussian noise with a standard deviation of 0.005), as the increase in the illumination angle reduces the response amplitude of the PTF [see Figs. S4(a)–S4(c) in Ref. [30]], the sensitivity to noise instead leads to an escalation in the RMSE, as illustrated in Figs. 3(d1)–3(d3). Although the annular NA-matched illumination has a larger illumination angle compared to the circular illumination with of 0.4 and 0.75, it has a relatively smaller RMSE in the presence of noise [Fig. 3(d4)], owing to its improved spatial frequency response that allows for higher robustness to noise [see Figs. S4(e) and S4(f) in Ref. [30]]. Essentially, NFTPM can be regarded as an iterative process that simultaneously seeks the optimal solutions for defocus distance prediction and phase retrieval. As shown in Figs. 3(f1)–3(f4), the loss function exhibits a steady decline along with a converging trend of defocus distance (towards 7 μm in all cases), indicating the parallel optimization of the model parameters and the defocus parameter. Additionally, simulations under annular illumination for various (5 μm, 8 μm, 12 μm, and 17 μm) in Fig. 3(g) validate the stability of NFTPM for defocus distance prediction.
Figure 3.Comparison between AI-TIE and NFTPM with an incorrect initial value of the defocus distance in various illuminations. (a) Intensity images under circular illuminations (, 0.40, and 0.75) and annular NA-matched illumination. (b) The results of AI-TIE. (c) The results of NFTPM. (d) The results of NFTPM in the presence of noise. (e) Ground truth. (f) Convergence curves of the defocus value and loss value (in logarithmic form). (g) The defocus distance prediction process of NFTPM for intensity images simulated at other defocus distances under annular illumination.
In the actual experiment, the long-time imaging of living HeLa cells [Fig. 4(a)] was performed using the inverted microscope (IX83) without motor drive adjustment or manual correction. Under the influence of temperature fluctuation and other factors that lead to focal drift, NFTPM shows its stability of phase recovery by correctly predicting the unknown defocus distance. Figure 4(b) demonstrates the iterative process of predicting the defocus distance using the NFTPM in Area 1 and Area 2 at different moments. Taking 1 μm as the initial value, we recovered the phase of the first frame measurement by NFTPM with the predicted defocus value of 6.522 μm and adopted 6.5 μm as the defocus distance for AI-TIE. Since the defocus distance was fixed at all moments, the retrieved results of AI-TIE, as depicted in Figs. 4(c1) and 4(c2), gradually deteriorated over time due to the model mismatch induced by the time-varying defocus distance. On the contrary, NFTPM dynamically reconstructed the phase information at different moments through adaptive defocus parameter correction, revealing distinct subcellular details such as nuclei and lipid droplets [Figs. 4(d1) and 4(d2)]. Once NFTPM reconstructs the phase for the measurement at a given moment using random initialization, the QPI of subsequent frames can be accelerated by initializing NFTPM with the MLP model and the predicted corresponding to the present frame, which utilizes the correlation between frames in the same FOV. Additionally, the efficiency of full FOV phase retrieval can be improved fivefold by utilizing the pre-iterated model of the subregion to initialize the neural field.
Figure 4.Experimental observation of HeLa cells via NFTPM under annular NA-matched illumination (see Visualization 1). (a) The full FOV of the reconstruction result of NFTPM. (b) The defocus distance prediction process. (c1), (c2) The phase retrieved by AI-TIE. (d1), (d2) The phase retrieved by NFTPM.
In summary, we have proposed a new partially coherent QPI method called NFTPM using the neural field. NFTPM, a single-shot non-interferometric iterative method, employs a straightforward MLP model for continuous phase representation and can accurately predict the defocus distance without prior knowledge, which eliminates the necessity for precise motor drive adjustment or manual correction for focus drift.
The rough defocus distance provided by the focusing device can be manually adjusted to reduce the WOTF error and thus improve the reconstruction results of AI-TIE. But with the limited time and manpower required, AI-TIE still cannot be applied as an effective method for long-term live cell imaging. In contrast, NFTPM replaces the costly manual operation with gradient-based tuning, which is based on the backpropagation algorithm in the prior model and the MLP. Therefore, NFTPM can adaptively obtain reconstruction results without defocus artifacts.
For circular illumination, high-resolution reconstruction results from large-angle illumination, providing more high-frequency information. However, this comes at the expense of a steadily diminishing PTF response with increasing illumination angle, making noise more detrimental to the phase reconstruction of NFTPM. In contrast, annular illumination exhibits strong noise immunity due to a uniformly high response in its pass-band over a large illumination angle. To obtain high-quality QPI results, the defocus distance also needs to be selected appropriately. The low-frequency response of the PTF becomes weak when the defocus distance is too small, which is not conducive to the recovery of low-frequency information. Besides, the PTF obtained at excessive defocus distance has a low response in its pass-band and contains multiple deep dips and zero-crossings, rendering this part of information susceptible to noise (see Figs. S5 and S6 in Section 4 of Ref. [30]). Remarkably, NFTPM can also be applied in pixel-aliasing conditions by additionally introducing pixel binning as a prior. Its capability of pixel super-resolution QPI is validated in Fig. S7 (Section 5 of Ref. [30]) by simulating a pixel-aliased defocused intensity of a USAF resolution test target.
Although our method incorporates partially coherent illumination into the forward image formation model, other optical parameters that are beneficial for improving reconstruction quality are still overlooked. Therefore, in the future, more optical parameters will be considered in the physical model to further promote the quality of phase reconstruction. For instance, it is possible to achieve prediction of unknown illumination by a grid search in a preset series of coherence parameters.
[1] G. Popescu. Quantitative Phase Imaging of Cells and Tissues(2011).
[18] D. Ulyanov, A. Vedaldi, V. Lempitsky. Deep image prior. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 9446-9454(2018).