1. INTRODUCTION

Digital holographic microscopy (DHM) is a powerful quantitative phase imaging technique [1,2]. Due to its noncontact, label-free and full field nature, it has found various applications: such as engineered surface topography measurement [3,4], biological cell imaging, [5,6], and so on [7]. With the development of optical setups and advancement of reconstruction algorithms, DHM is ongoing, and techniques that exceed current spatial and temporal resolution, achieving high phase sensitivity and high throughput imaging capabilities, have emerged [8–10]. The advancement of DHM has in turn boosted its application in high fidelity quantitative phase imaging for both biomedicine and industrial metrology. In these applications, the behaviors and morphologic features of a cell or a neuron in different biological processes can be monitored [10], or topography of a micro-device during its precision manufacturing can be inspected [9].

In DHM, an objective is used to magnify the measured sample. The introduction of the objective would introduce a spherical phase curvature in the object beam. In contrast, usually the reference beam is set as a plane beam. The wave front mismatch between the two beams will form the main part of the phase aberration in the measured phase map [11,12]. In DHM, the optical path difference (OPD) induced by a specimen is recorded in the form of a hologram. However, during the recording, the unwanted phase terms, i.e., phase aberrations, are also recorded simultaneously, besides the desired phase map associated with the specimen. The phase aberrations generally include a linear phase difference introduced by the off-axis interferometric geometry, the spherical phase curvature caused by the microscopic objective, and other high-order phase terms related to the optical lens [13]. These phase aberrations will superimpose on the sample’s phase to block the identification of the sample; thus, it needs to be compensated to get the exact sample phase image. On top of this, the phase image of the specimen also suffered from the coherent noise [14]. Due to the highly coherent laser used in the setup, parasitic interference fringes resulting from multiple reflections from the coverslip and diffraction patterns from the dust or scratches on the surfaces of the lens are introduced into the hologram, thus resulting in phase noise in the reconstructed image. Finally, in the reconstruction procedure, both the phase aberrations and the coherent noise were superimposed on the sample phase map, thus seriously degrading the phase fidelity and hampering its application of DHM in phase imaging. An obvious advantage of DHM is its post-processing capability. With this, the phase aberrations and coherent noise in the phase image could be processed numerically. To achieve accurate phase imaging, a general reconstruction procedure containing multiple steps is shown in Fig. 1(a). The procedure includes two independent phase aberration correction and phase denoising processes, besides the basic wrapped phase extraction and phase unwrapping operations.

 

Fig. 1. Schematic diagram of phase aberration compensation and denoising. (a) Conventional method and (b) our method.

Download Full Size | PDF

In terms of phase aberration correction, various numerical methods have been suggested, and they can be classified into two groups: traditional numerical compensation [15–26] and learning-based methods [13,27–32]. With the traditional numerical compensation methods, the phase curvature can be estimated by the least-square fitting method with either polynomial functions [16–18] or Zernike polynomials [19]. Phase aberrations that contain only linear and spherical phase terms can be compensated with the principal component analysis (PCA) method [20], spectral analysis method [12], or geometric transformation method [21,22]. The phase aberrations can also be evaluated by nonlinear optimization, but with a heavy computation burden [23–25]. Recently, a phase variable splitting framework based on alternating direction sparse optimization for aberration correction was proposed [9,11,26]. Faithful phase reconstruction is demonstrated both in DHM [11] and in synthetic-aperture phase microscopy [26]. The deep learning-based methods have emerged as a new kind of technique, demonstrating outstanding performance in aberration correction. With a convolutional neural network (CNN) Unet, the background in the phase map can be segmented, and the phase aberration is evaluated followed with a polynomial fitting operation [27]. The aberration-free phase image can also be output directly with a trained Unet [28]. The phase aberration can be estimated by determining the coefficients of its Zernike polynomials by means of linear regression with a ResNet-50 model [13] or a self-supervised sparse constraint network (SSCNet) [29], and the reconstruction accuracy could be further improved by implementing an additional numerical fitting procedure [30]. Instead of processing the phase image directly, a sample-free hologram is generated with a trained two-stage generative adversarial network (GAN) from a masked hologram where the sample area is labeled in advance [31], or directly from a sample hologram [32].

On the other hand, coherent noise presented in the phase image can be reduced either by digital image processing methods [3,14,33] or learning-based methods [34–37]. The median filtering may be a basic operation that suppresses the noise to some extent, but the finer details of the sample may also be smoothed. The low-pass filtering simply assumes the noise is in the high-frequency region in the spectrum domain, and a Butterworth digital filter or a Gaussian filter can be designed to filter out the high-frequency components [3]. Block-matching and 3D filtering (BM3D) is a state-of-the-art technique that combines non-local filtering and transform domain filtering and demonstrates impressive denoising performance among the traditional digital methods [14,33]. Deep learning-based methods can also be used to suppress the noise in the phase image. The speckle noise in a phase map of a deformed surface for stress analysis can be suppressed with a retrained DnCNN model, which is originally designed for Gaussian noise removal in the natural image [34]. The same kind of noise can also be reduced with a GAN model with a relative smaller dataset for training [35]. Without the noise-free image, a self-supervised network can be trained with a dataset containing only a noisy image for denoising an image [36]. Better performance is achieved with a DnCNN model incorporating the attention mechanism and residual connection which are trained with numerically simulated datasets [37]. However, in these learning-based denoising methods, to generate the training data, the noise in the image is simply considered as speckle noise [34,35], Gaussian noise [36] or Perlin noise [37], while the coherent noise in the phase image is so complicated that it cannot be characterized by any simple model [14]. As a result, the network may be effective in dealing with the same kind noise that is included in the training dataset, while other kinds of noises remain in the final phase map [37].

Currently, the problems of phase aberration correction (PAC) and denoising (DN) are treated as two independent tasks that are executed sequentially in the regime of phase imaging with DHM, for example, the execution of PAC in advance followed by DN, as shown in Fig. 1(a). However, noises in the phase image seriously affect the efficiency of the PAC procedure, especially when the least-square based fitting method is adopted [16–19]. Alternatively, DN can be applied before PAC. Aberrations presented in the phase image make it a tricky task to denoise. Inadequate filtering results in residual noise in the phase image, while excessive filtering leads to over-smoothing of the sample. Here, for the first time to the best of our knowledge, we present a one-step method that achieves PAC and DN simultaneously by using deep learning. A simulated dataset with a wrapped phase image as network input is synthesized to train a CNN. With the trained CNN, as shown in Fig. 1(b), an image-to-image transformation from an aberrated, noisy wrapped phase image to aberration-free , noise reduced phase image is directly produced. With this one-step method, not only is the reconstruction process in phase imaging simplified, but also the performance of the trained CNN is proved to be state-of-the-art in PAC and DN.

2. METHODS

A. Aberrated and Noisy Phase Data

In DHM, a hologram is formed by coherent addition of a complex object wave $O(x,\;y)$ and a reference wave $R(x,\;y)$. In the image sensor plane, its intensity can be expressed as

Here the first two terms are the direct components, the third term contains the complex object wave, the fourth term contains its conjugate, and the fifth term contains the noises which include shot noise, speckle noise, and parasitic fringes, etc. [14].

In an off-axis hologram, as carrier frequency is presented, the complex object wave can be retrieved with the spectrum filtering method [4], and it is denoted as $U(x,\;y)$. In the reconstruction procedure, a wrapped phase image is obtained subsequently:

To remove the modulo ${2}\pi$ in $\varphi (x,\;y)$, a phase unwrapping algorithm is applied to get the unwrapped phase image $\phi (x,\;y)$. However, in addition to the desired phase image ${\phi _o}(x,\;y)$ of the specimen, it contains both phase aberrations ${\phi _a}(x,\;y)$ and phase noises ${\phi _n}(x,\;y)$. Hence, the unwrapped phase image $\phi$ is expressed as

Conventionally, to get the accurate phase image ${\phi _o}(x,\;y)$ two independent steps are used to remove the phase aberrations and phase noises. Conversely, in the proposed method, a wrapped phase image is put into the CNN, and the desired phase image ${\phi _o}(x,\;y)$ is output directly.

 

Fig. 2. Data preparation process.

Download Full Size | PDF

To train the CNN, a large amount of wrapped phase images is generated according to Eq. (3). The aberration phase terms could be modeled with a parabolic function [12], standard polynomials [18], or Zernike polynomials [19]. Here, for variousness of the phase maps and simplicity in generating training data, ${\phi _a}(x,\;y)$ that include linear phase tilt, quadratic phase curvature, and high-order aberrations, are generated with Zernike polynomials:

Here ${Z_i}$ is the $i$ th-order Zernike polynomial, and ${c_i}$ is the Zernike coefficient. The 2nd and 3rd Zernike terms represent linear phase delay along $x$, $y$ directions, respectively. The 4th term denotes defocus. These tree terms constitute the main aberration in ${\phi _a}(x,\;y)$. Thus, to generate diverse fringe patterns, the 2nd and 3rd Zernike’s coefficients are set in the range of [1,120], and the 4th term is in the range of [${-}{25}$, 25], while other terms are [${-}{0.5}$, 0.5].

Secondly, phase noise ${\phi _n}(x,\;y)$ is acquired through experiments. In previous deep learning-based denoising methods, phase noise is added by simulating Gaussian noise or speckle noise [34–36]. Nevertheless, these regular types of noises cannot represent the actual sophisticated coherent noises in experimental conditions, while the performance of trained CNNs depends on the noise type in the training set. Here, to enhance its denoising capability for actual phase noises, a set of phase noise images was collected by using the double exposure method [38]. The double exposure method is a reliable method capable of eliminating both low-order and high-order phase aberrations. However, it requires two holograms: one with a sample and the other without a sample. In an off-axis digital holographic microscope, a blank slide is placed at the stage, and a hologram is recorded. A second hologram is recorded by moving the slide a little. With the spectrum filtering method, a phase image containing only noise is obtained from these two holograms. During the experiments, we shifted different objectives and selected different slides. We also slightly adjusted the illumination beam angle and the interference angle to produce various phase noises. Totally, 50 high-resolution phase images containing only coherent noises were reconstructed with the double exposure method, and they were used for generating ${\phi _n}(x,\;y)$.

Finally, the sample phase image ${\phi _o}(x,\;y)$ is obtained through both the simulation and experiments. A part of the samples that has binary patterns is generated with the simulation. A set of high-resolution target images containing binary patterns, such as Ronchi gratings with different periods and different ruling directions and an NBS-1963A image, are prepared. The simulated sample phase maps are then generated from these images. A region of ${256} \times {256}$ pixels is randomly cropped from one of the high-resolution images. Then it is multiplied with a random value in the range of (0, ${2}\pi$] to simulate the height of the sample and serve as a sample phase map. The other part of the samples is prepared experimentally. We measured a set of blood smears and PMMA beads in suspension oil. These sample phase maps are first reconstructed with the double exposure methods [38] from which the phase aberrations were compensated. To remove the coherent noises in the images, the BM3D [33] which is a state-of-the-art denoising method, was applied. The filtered images are considered as the accurate (ground truth) sample phase images. Similarly, a ${256} \times {256}$ pixel region is randomly cropped from a filtered image to generate training data.

 

Fig. 3. Training loss and validation loss curves for two networks: (a) Unet3+ network and (b) PACUnet3+ network.

Download Full Size | PDF

By adding an aberration term and noise phase term to the sample phase map, the noisy and aberrated phase maps are obtained. A modulo ${2}\pi$ operation is then applied to get the wrapped phase map $\phi$ as the input data for training the CNN. This data generation process is schematically shown in Fig. 2. In addition, in order to further increase its adaptation of the CNN to practical data, 300 wrapped phase maps reconstructed directly from experiments were also added into the dataset. Totally, a dataset containing 30000 data pairs (wrapped phase image $\phi$ and noise-free wrapped phase image ${\phi _o}$) is generated. It was divided into a training dataset and a validation dataset in the ratio of 8:2.

B. Network Training

The CNN used in this work is PACUnet3+ which has been proposed recently [32]. PACUnet3+ derives from the original Unet3+ network and inherits its structure. The network consists of two parts: encoders and decoders, where a full-scale skip connection is adopted to make full use of multiple-scale feature maps to improve the training accuracy [39]. Instead of a simple convolution layer as used in Unet3+, in each layer of both encoders and decoders, a residual connection block (ResBlock) [40] is adopted to prevent network degradation and reduce the loss of features information. In the meantime, the efficient channel attention (ECA) mechanism [41] is implemented in the ResBlock to enhance channel characteristics. In previous work, the PACUnet3+ is trained to predict a sample-free hologram from a sample hologram. The phase aberration is then corrected such as that in the double exposure method. However, the input hologram should have the same size as that in the training. Resizing an off-axis hologram having a bigger size than that in training would induce an aliasing effect such that the network may fail to work.

Here a wrapped phase image reconstructed from an off-axis hologram is put into the network, and the output provides the sample phase image directly. In the training, 24000 wrapped phase images are used as input, while the corresponding sample phase images are used as training labels. The other parts of data pairs are used for validation. The root mean square error (RMSE) is one of the most used metrics to calculate the difference between two images, while the structural similarity index measure (SSIM) is another metric evaluating the similarity between two images. During the training, we tried and found a hybrid lost function that can be continuously minimized with the Adam optimizer:

The initial learning rate was ${1{\rm e} - 4}$, and it was reduced to one-half of its previous value every 25 epochs. A batch size of 8 that was set due to the network has a large amount of parameters to be updated and the limitation of the training platform. The network was trained with 110 epochs in a server with core ${12} \times$ Xeon Platinum 8260 CPU and 84 G of ram, using NVIDIA GeForce RTX 3090. The whole training time lasted for 150 h, and it takes about 0.75 s for the trained network to output an aberration- and noise-free phase map from the input. For comparison, Unet3+ was also trained with the same dataset. Figure 3 presents the training loss and validation loss curves for both networks. After the same training epochs, PACUnet3+ always has the smaller training loss values than Unet3+. In addition, the validation curve of Unet3+ fluctuated seriously. From these we know that PACUnet3+ would have a better performance than its counterpart Unet3+.

3. NUMERICAL RESULTS

After training, we first compared the performance of PACUnet3+ against Unet3+. In order to evaluate the accuracy of the network output, the point-to-point phase error map is calculated by subtracting the simulated ground truth sample phase map from the output. The RMSE and the peak-to-valley (PV) are two quality metrics to analyze the performance of PAC and DN for the network. In addition, to further evaluate the DN capability of the network, the peak signal-to-noise ratio (PSNR) of the output image is calculated where the ground truth sample phase map is used as the reference. We generated 100 new image pairs, i.e., wrapped phase images containing the aberrations and noise-free wrapped sample phase images, according to section 2.1. The 100 aberrated and noisy wrapped phase images were used as input to the trained networks. Subsequently, two groups of 100 phase error maps corresponding to two networks were calculated. The statistical results of the RMSE and PV of these two groups of error maps are shown in Figs. 4(a) and 4(b), respectively. For PACUnet3+, the average RMSE value is 0.0654 rad, and the average PV value is 0.967 rad, while they are 0.1359 rad and 1.5062 rad with Unet3+. These values indicated that PACUnet3+ has a better performance than that of the Unet3+ network. In addition, the distribution of the PSNR for two groups of outputs are presented in Fig. 4(c). The average PSNR value is 30.623 dB for the PACUnet3+, while it is 27.108 dB for Unet3+; the average PSNR for the original maps is 16.712 dB, which further indicated that both networks could filter the noise efficiently, and the trained PACUnet3+ has a better denoising capability.

 

Fig. 4. Statistical results based on 100 simulated wrapped phases with PACUnet3+ and Unet3+. (a) RMSE values for point-to-point deviation, (b) PV values for point-to-point deviation, and (c) PSNR values for outputs.

Download Full Size | PDF

To intuitively demonstrate the reconstruction results, two different samples are shown in Fig. 5. For both examples, Fig. 5(a) shows the wrapped phase images, where phase aberrations and phase noise are presented. Figure 5(b) shows two sample phase maps which are aberration-free and noise-free. The outputs predicted by the trained PACUnet3+ are shown in Fig. 5(c). As a comparison, Fig. 5(d) presents the results output by Unet3+. The profiles along the lines marked in Figs. 5(b)–5(d) are also plotted in Fig. 5(e) to compare the reconstruction accuracy.

 

Fig. 5. Compared reconstruction results of two simulated wrapped phase images, the first example (I) and the second example (II). In both cases: (a) noisy wrapped phase image, (b) ground truth sample phase, (c) output from PACUnet3+, (d) output from Unet3+, and (e) crossing profiles along lines shown in Figs (b–d).

Download Full Size | PDF

For comparison, these two samples are also constructed with three other conventional methods. First, the phase aberrations presented in the wrapped phase images that are shown in Fig. 5(a) are compensated with the PCA, Zernike polynomial fitting (ZPF), and traditional 2D polynomial fitting (PF) methods. Subsequently, the three aberration corrected maps are filtered with BM3D to reduce the phase noise, and the final results are showcased in Fig. 6. Figure 6(a) presents the results reconstructed from the PCA and BM3D methods. Due to the fact that the PCA cannot deal with high-order phase aberrations, the results in Fig. 6(a) have obvious errors. With the other two fitting methods, the phase aberrations have not been fully compensated, as shown in Figs. 6(b) and 6(c). A possible reason is the phase noise has an adverse effect on numerical fitting. Figure 6(d) also plots the profiles along the same lines marked in Figs. 6(a)–6(c). The evaluation indices of the reconstructed results are also calculated and presented in Table 1. From the results presented in Figs. 5 and 6 and Table 1, we know that the results obtained from PACUnet3+ have fewer reconstruction errors and higher SNRs than Unet3+ and the other three conventional two-step methods.

 

Fig. 6. Reconstructed phase images with conventional two-step methods. In both cases: (a) result obtained by the ${\rm PCA} + {\rm BM3D}$ method, (b) result obtained by the ${\rm ZPF} + {\rm BM3D}$ (c) result obtained by the ${\rm PF} + {\rm BM3D}$, and (d) crossing profiles of phase images along lines shown in (a–c).

Download Full Size | PDF

4. EXPERIMENTAL RESULTS

To experimentally demonstrate the capability of the network, we tested two different samples: a USAF 1951 resolution target and a blood smear. The off-axis holograms were recorded in a transmissive digital holographic microscope that are based on a Mach–Zehnder interferometer; the detailed description of the setup can be found in Ref. [32]. In the setup, a tube lens is placed behind the microscopic objective to compensate for most of the phase aberrations, which is a general means for most of the digital holographic microscopes [27,28]. For each of the samples, the off-axis hologram is first Fourier transformed, and its object diffraction order is filtered. Afterwards, both wrapped phase maps are reconstructed and presented in Fig. 7(a). For comparison, these two samples are also constructed with the double exposure method to compensate for the phase aberrations and denoised with the BM3D method to suppress the coherent noise. These two phase maps are considered to be aberration-free and noise-free. They are presented in Fig. 7(b), and they are used as the references for comparison. Though the double exposure methods can produce an aberration-free phase image, it needs to record two holograms: a sample hologram and a sample-free hologram. However, the sample-free hologram is not always available. In contrast, the suggested method only needs a sample hologram to retrieve a wrapped phase map as the input of the network. The results predicted by PACUnet3+ and Unet3+ are shown in Figs. 7(c) and 7(d), respectively. For comparison, three indices of the results are calculated, and they are listed in Table 2. For the first example, the phase error of the result obtained from PACUnet3+ has a RMSE of 0.1044 rad and a PV of 2.3943 rad. As a contrast, they are 0.1467 and 2.8651 rad for the result obtained from Unet3+. In addition, the result of PACUnet3+ has a higher PSNR value of 29.7463 dB than that of 25.154 dB obtained from Unet3+. For the second sample, similar results are obtained, and they are listed in the figure and Table 2. It should be noted that the phase maps reconstructed from the double exposure method shown in Fig. 7(b) have obvious phase ripples or phase fluctuations in their backgrounds. Such phenomena may be induced by the parasitic fringes or serious speckle noises recorded in the hologram, which could not be fully filtered by any of a the conventional denoising methods. In contrast, the phase maps predicted by deep learning-based methods have relative smoother backgrounds. It should be noted that the processed phase images here have a resolution of ${1280} \times {960}$ pixels, which is much bigger than that in training. To process these phase maps, they are resized to a resolution of ${256} \times {256}$ and then fed into the trained network. The output images from the network are enlarged accordingly. This is an obvious advantage of this method, as different sizes of the wrapped phase image due to different camera sensors can be processed at once. Conversely, in other deep learning methods with a hologram as its input, downsampling a larger hologram to the same size as that in the training generally leads to aliasing and thus results in failure of the methods [31,32].

We also reconstructed the sample with three other conventional methods where two independent steps are applied for phase aberration compensation and denoising. The reconstructed results are presented in Fig. 8, and the calculated evaluation indices are listed in Table 2. For both samples, none of the three methods, ${\rm PCA} + {\rm BM3D}$, ${\rm ZPF} + {\rm BM3D}$, or ${\rm PF} + {\rm BM3D}$, can fully compensate for the phase aberrations and remove phase noises. It should be noted that in the last two methods, before phase fitting, a phase unwrapping procedure is applied to get the unwrapped phase map; the excessive coherent noise may induce phase errors in this step and thus affects the accuracy of numerical fitting. For both samples, phase profiles along the lines in different reconstructed results are also plotted in Fig. 8(d). It is obvious that phase curves obtained from conventional methods exhibit severe deviations when compared with the reference phase curve. Among the results presented in Figs. 7 and 8, the phase curve of PACUnet3+ matched best to the reference. The presented results indicate that for the wrapped phase maps obtained from experiments, the trained PACUnet3+ is less affected by the complicated phase aberrations and coherent noises and capable of yielding a sample phase image with the highest accuracy.

Table 1. Comparison of Evaluation Indices between Different Methods for Simulated Samples

View Table | View all tables in this article

 

Fig. 7. Compared reconstruction results of two experimental wrapped phase images. In both cases: (a) noisy wrapped phase image, (b) reference phase image, (c) sample phase image from PACUnet3+, (d) sample phase image from Unet3+, and (e) crossing profiles along lines shown in (b–d) for comparison.

Download Full Size | PDF

 

Fig. 8. Reconstructed phase images with conventional two-step methods. In both cases: (a) result obtained by the ${\rm PCA} + {\rm BM3D}$ method, (b) result obtained by the ${\rm ZPF} + {\rm BM3D}$, (c) result obtained by the ${\rm PF} + {\rm BM3D}$, and (d) crossing profiles along lines shown in (a–c) for comparison.

Download Full Size | PDF

5. CONCLUSION

In summary, a one-step state-of-the-art deep learning-based algorithm can be trained for PAC and DN. For achieving this, a faithful database is constructed for training according to the wrapped phase map generation model. In the database, the phase aberration containing both low- and high-order aberrations are simulated with Zernike polynomials, while phase noises are from those recorded in real measurement setups. Thus, the trained CNN has a good adaptability to practical aberrated and noisy phase maps. With the trained CNN, an aberration-free and denoised phase map is yielded directly from a wrapped phase map. Both the simulation and experimental results show that the proposed method can output a sample phase map with lower RMSE and PV values and a higher PSNR value when compared with the Unet3+ network and the other three conventional two-step methods. Except for its high-accuracy phase aberration compensation and phase noise removal capability, the proposed method could be an efficient way to optimize the reconstruction pipeline for quantitative phase imaging in DHM due to its two step in one nature.