Opto-Electronic Advances, Volume. 6, Issue 4, 220105(2023)

Deep learning enhanced NIR-II volumetric imaging of whole mice vasculature

Sitong Wu1,2、†, Zhichao Yang1,2、†, Chenguang Ma1、†, Xun Zhang1, Chao Mi1, Jiajia Zhou2, Zhiyong Guo1,3、*, and Dayong Jin1,2,3、**
Author Affiliations
  • 1UTS-SUSTech Joint Research Centre for Biomedical Materials & Devices, Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen 518055, China
  • 2Institute for Biomedical Materials & Devices, Faculty of Science, University of Technology Sydney, Ultimo, New South Wales 2007, Australia
  • 3Guangdong Provincial Key Laboratory of Advanced Biomaterials, Southern University of Science and Technology, Shenzhen 518055, China
  • show less

    Fluorescence imaging through the second near-infrared window (NIR-II,1000–1700 nm) allows in-depth imaging. However, current imaging systems use wide-field illumination and can only provide low-contrast 2D information, without depth resolution. Here, we systematically apply a light-sheet illumination, a time-gated detection, and a deep-learning algorithm to yield high-contrast high-resolution volumetric images. To achieve a large FoV (field of view) and minimize the scattering effect, we generate a light sheet as thin as 100.5 μm with a Rayleigh length of 8 mm to yield an axial resolution of 220 μm. To further suppress the background, we time-gate to only detect long lifetime luminescence achieving a high contrast of up to 0.45 Ιcontrast. To enhance the resolution, we develop an algorithm based on profile protrusions detection and a deep neural network and distinguish vasculature from a low-contrast area of 0.07 Ιcontrast to resolve the 100 μm small vessels. The system can rapidly scan a volume of view of 75 × 55 × 20 mm3 and collect 750 images within 6 mins. By adding a scattering-based modality to acquire the 3D surface profile of the mice skin, we reveal the whole volumetric vasculature network with clear depth resolution within more than 1 mm from the skin. High-contrast large-scale 3D animal imaging helps us expand a new dimension in NIR-II imaging.

    Introduction

    Thanks to the substantial reductions of tissue scatterings and absorption when fluorophores emit at a longer wavelength1, fluorescence imaging through the second near-infrared window (NIR-II: 1000~1700 nm) has been widely developed, which includes both a series of NIR-II contrast agents2-4 and a range of advanced imaging systems. The recent advances made in volumetric imaging systems can be compared in a pyramid of resolution, speed, depth, and volume of view. In 2010, Welsher K et al. proved that the penetration depth can be maximized when choosing the working wavelengths between 1 and 1.4 µm5. In 2012, Hong et al. developed single-walled carbon nanotubes for NIR-II fluorescent imaging to achieve a high spatial resolution of ~30 µm, fast acquisition of less than 200 ms, and satisfactory tissue penetration of up to 3 mm6. In 2014, the group made progress in through-skull fluorescence imaging of mice without craniotomy and demonstrated an imaging depth of more than 2 mm with a resolution better than 10 µm, a field of view of 25 mm × 20 mm, and a frame rate of 5.3 per second7. In 2019, Wenbin et al. developed a confocal microscopy system using NIR-II aggregation-induced emission dots, achieved an 800 µm-deep 3D in vivo cerebrovascular imaging of a mouse, and reached a spatial resolution of 8.78 µm at 700 µm depth8. Liu Y et al. reported the use of carbon quantum dots for NIR-II two-photon bioimaging with a maximum penetration depth of 500 µm9. In 2021, Wang et al. reported structured-illumination light-sheet microscopy with a penetration depth up to 750 µm and a spatial resolution of 1.9 µm × 1.9 µm × 1.9 µm10. With the four key performance specifications of the field of volume, imaging speed, depth resolution, and lateral resolution being quantitatively summarized in the schematic Fig. 1, the trade-off becomes obvious among the choices of three light illumination strategies, wide-field, spot scanning, and light-sheet scanning.

    Trade-offs among lateral resolution, depth resolution, speed, and volume of view by using the three light illumination strategies, wide-field11, spot scanning12, and light-sheet scanning13, in NIR-II fluorescent volumetric imaging (log scale).

    Figure 1.Trade-offs among lateral resolution, depth resolution, speed, and volume of view by using the three light illumination strategies, wide-field11, spot scanning12, and light-sheet scanning13, in NIR-II fluorescent volumetric imaging (log scale).

    Light-sheet fluorescent microscopy, by illuminating a single optical section at a time, makes volumetric imaging practical with reasonable spatial and temporal resolutions14. The thin illumination layer minimizes the phototoxicity effect15, and z-scanning with an array detector reconstructs the 3D images. In 2021, Wang et al. combined light-sheet microscopy with a type of lanthanide-doped NIR-II fluorescent nanoparticles to reduce out-of-focus background and improve spatial resolution10.

    There are several challenges towards extending the light sheet system for vasculature imaging at millimeter-scale in each dimension. First of all, there is a trade-off between the length and the thickness of light sheet ruled by the Gaussian beam distribution of laser. Moreover, the scattering effect prevents high-contrast imaging in millimeter-scale tissue, as the surrounding area can be also excited by the strongly scattered light through the thick tissue. The spectrum filters may not completely shield the laser scatterings and background. Consequently, the low contrast prevents the detection system from resolving vasculature's fine details.

    Time-gated (TG) fluorescence imaging has attracted increasing attention16, 17. It separates long-lifetime fluorescence by gating off the pulsed laser excitation. Theoretically, this method can completely remove the laser scattering light and autofluorescence toward high contrast16. In 2019, Gu et al. developed a new contrast agent of Yb3+ doped nanoparticles, which has near-unity quantum yield and sufficient long lifetime at NIR so that time-gated imaging can be used to achieve a high signal-to-noise ratio of >9 under an excitation power density of 1.1 mW/cm 218. In 2020, B Nimmegeers et al. prepared NIR emitting nanoparticles for time-gated imaging to remove the autofluorescence from the biological tissues19. To implement the scheme of time gating in the visible range, we can use a modulated camera or a switchable single element detector20-22. However, when working at the NIR-II, a time-gatable camera needs to be custom-made with high cost. Using a chopper to switch the light path16, 23, 24 is a low-cost and practical strategy.

    Computational imaging, by developing algorithms for data analysis, can significantly improve the imaging quality. For example, a vessel enhancement algorithm can facilitate to identify vessel networks from the interference of background tissue, which is usually based on the segmentation of blood vessels by imaging gradient, intensity, and texture. Hessian matrix-based method calculates the partial derivative to segment the vessel according to the mutation gradient25. Principal component analysis (PCA) based method extracts vessel signals by removing the background5. However, both methods can be affected by the background gradient. In this work, we removed the most influence from the background gradient and developed an algorithm based on deep learning to segment the vessel.

    Results and discussion

    To generate a light sheet, we set the focal spot on the designed distance to the back focal plane of the objective lens to form a lensset with low Numerical Aperture (NA). We adjust the focal spot using an objective lens (10×, NA=0.3, Nexscope planF-Apo) and reshape the Gaussian beam by a light sheet using a Powell lens array (PMMA-Powell lens array-30). Powell lens array provides more uniform sheet illumination than cylindrical lens26. The FWHM (full width at half maximum) of the light sheet can be calculated by:

    2ω0=2nλπNA,

    where λ is the wavelength,n is the refractive index. Using the Gaussian beam model, we get the diameter and length of the beam waist, the Gaussian beam functions are shown as:

    E=E0ω0ω(z)exp(r2ω2(z)),

    ω(z)=2ω0,

    z0=ω2πηλ,

    where E is the amplitude of the light beam, E0 is the amplitude at the focal plane, z is the distance from the focal plane,r is the distance from the optical axis, ω(z) is the spot radius,ω0 is the spot radius at the focal plane. z0 is the Rayleigh length,η is the refractive index of the medium,λ is the wavelength. The Rayleigh range is the range when the spot area becomes twice the minimum value, and the Rayleigh length of a Gaussian beam is the length from the waist to the spot of the Rayleigh range.

    The surface of the mice is not flat, and therefore sufficient Rayleigh length is required for light-sheet scanning. To achieve a sufficient length of light-sheet illumination, we chose the Rayleigh length as 8 mm, so that the simulated radius of the focal spot is calculated as 100 µm in diameter. In our experiment, we achieved the focal spot with a diameter of 100.5 µm when the Rayleigh length was about 8 mm, as shown in Fig. 2. Towards a deeper illumination with the Rayleigh length of up to 26 mm, the thickness of the light sheet can be adjusted to 183.1 µm when adjusting the distance between the lens set and the objective. The comparison results of wide-field imaging and light-sheet imaging (maximum projection) are shown in Fig. 2(c, d). The result indicates that light-sheet imaging provides lower background intensity. The light sheet result shows a better contrast ratio than wide-filed illumination because of less scattering. With light-sheet mode illumination, there is only scattering light in the light sheet area. However, large surrounding scattering light overlay together, leading to a high-intensity background.

    Adjustable large-FoV light-sheet illumination system for high-contrast whole mice imaging. (a) light-sheet illumination system with adjustable thickness from 183.1 to 100.5 μm and Rayleigh length from 8 to 26 mm, formed by a 30 mm focus lens (Thorlab-1700-B), a 16 mm focus 0.79 NA aspherical lens (Thorlab-ACL254U-B), and a 10 × 0.3 NA objective lens. (b) The illumination of light sheet through the whole mice. (c) Comparison of a wide-field imaging result and a top view of light-sheet scanning result at a distance of 100 μm from the mice. (d) Line profile analysis of (c).

    Figure 2.Adjustable large-FoV light-sheet illumination system for high-contrast whole mice imaging. (a) light-sheet illumination system with adjustable thickness from 183.1 to 100.5 μm and Rayleigh length from 8 to 26 mm, formed by a 30 mm focus lens (Thorlab-1700-B), a 16 mm focus 0.79 NA aspherical lens (Thorlab-ACL254U-B), and a 10 × 0.3 NA objective lens. (b) The illumination of light sheet through the whole mice. (c) Comparison of a wide-field imaging result and a top view of light-sheet scanning result at a distance of 100 μm from the mice. (d) Line profile analysis of (c).

    Though NIR-II light at longer wavelength leads to less optical scattering interaction with the tissue, it still shows a certain degree of unavoid scatterings. The scattered light from the laser excites the surrounding fluorescent material so that the obtained image looks thicker (reduced resolution) than expected (Fig. 3(a)). In Fig. 3(b–d), we evaluate the depth-dependent scattering effect of our light-sheet scanning system and visualize it via the 3D reconstruction of a pattern. We made four silicone phantoms containing 1550 nm-emissive downshifting nanoparticles (DSNPs, see Methods for the nanoparticle synthesis and morphology) and TiO2 powder, in which the former serves as the contrast agent and the latter as scattering media with varying concentrations from 0 to 0.8 mg/mL (Fig. 3(b)). We used a 980 nm light to illuminate the phantoms and imaged their side profile. As the TiO2 concentration increases, the scattering effect became severe. In the real tissue medium, the degree of scattering effect varies randomly. The distance-dependent scattering effect and the tissue section of interest will become hard to be quantified in a conventional small animal imaging system using wide-field illumination. Therefore, the resolutions in both lateral and axial directions become unpredictable. By contrast, in our case here, the application of light-sheet illumination improves our ability in optical sectioning, so that we can precisely locate the real position of the tissue section in a much small and confined area, due to the Gaussian distribution of light-sheet beam across the tissue. As the center of the light sheet has maximum power density where the fluorescent intensity reaches the maximum value, we can use the maximum position as the center of the blood vessel during the 3D reconstruction process. Here we embedded some cotton materials of 500 µm diameter into a silicone phantom to form a pattern of letters SUSTech stained with DSNPs. The letters are distributed at different depths through the phantom matrix (Fig. 3(c, d)). The 3D image reconstruction result matches the real structure of the phantom matrix, suggesting the accurate localization ability in our large-FoV imaging.

    Phantom experiment to evaluate the scattering effect under the large-FoV light-sheet illumination. (a) Schematic illustration of light-sheet scanning across the phantom matrix. (b) Scattering simulation using silicone phantom containing lanthanide doped NIR-II fluorescent nanoparticles and TiO2 nanoparticles. (c) Letter patterns stained by lanthanide-doped NIR-II fluorescent nanoparticles and embedded in silicone phantom. (d) Reconstruction of the letter patterns by light-sheet scanning. The diameter of the container is 5.5 cm. The step of light-sheet scanning between each layer is 1 mm.

    Figure 3.Phantom experiment to evaluate the scattering effect under the large-FoV light-sheet illumination. (a) Schematic illustration of light-sheet scanning across the phantom matrix. (b) Scattering simulation using silicone phantom containing lanthanide doped NIR-II fluorescent nanoparticles and TiO2 nanoparticles. (c) Letter patterns stained by lanthanide-doped NIR-II fluorescent nanoparticles and embedded in silicone phantom. (d) Reconstruction of the letter patterns by light-sheet scanning. The diameter of the container is 5.5 cm. The step of light-sheet scanning between each layer is 1 mm.

    Here in our system, the low scattering effect of light-sheet illumination leads to a high resolution in a deep depth. Moreover, the residue laser scattering can be removed by the time-gated detection by collecting the signal from the long lifetime DSNPs (2 ms, see Supplementary information Methods) after the pulsed laser excitation off, towards a high imaging contrast. As shown in Fig. 4(a), in the detection optical path of the system, we inserted a chopper in between the first lens with an 8 mm focus length and the second lens with a 25 mm focus length to realize the time-gating. The detection path only opens when the laser excitation is switched off so that any short-lived background becomes negligible. The chopper used here (C995, Terahertz Technologies) is designed to have a blade consisting of 8 slots with a duty cycle of 2.23:1, making sure the most excitation light is detected. Though the chopper blades could remove all the excitation light, it causes part of the emission light to be lost. To maximize the luminous flux in detection, we optimize the excitation time, detection time, and detection frequency, according to the fluorescence lifetime and integral calculation (see Supplementary information Methods). When operating at a frequency of 250 Hz, it gave a rotational speed of ~31.25 rev/s. A 6-mm diameter pinhole aperture was attached very close to the chopper blade at a radius of 4.2 cm so that an ON/OFF switching time of 1856 µs was achieved for the signal light with any stray light being removed. The chopper outputs a TTL signal, generated from the slotted optical switch built in the chopper head, to trigger a homemade pulse synchronizer. The latter delivered pulses of 1858 µs duration to the laser controller/driver to switch on the 980 nm laser when the detection path was blocked by the chopper blade so that a SWIR camera became effectively time-gated.

    Time-gated light-sheet imaging system. (a) TG imaging system. (b) Comparison results of TG light-sheet imaging and CW filter-based light-sheet imaging with different average laser power. (c) Icontrast of TG mode and CW mode with different average laser power. (d) Range of intensity of TG mode and CW mode with different average laser power.

    Figure 4.Time-gated light-sheet imaging system. (a) TG imaging system. (b) Comparison results of TG light-sheet imaging and CW filter-based light-sheet imaging with different average laser power. (c) Icontrast of TG mode and CW mode with different average laser power. (d) Range of intensity of TG mode and CW mode with different average laser power.

    Figure 4(b) shows the time-gated (TG) imaging result in comparison with the continuous wave (CW) mode with average laser power from 54 mW to 486 mW. The high peak power of TG mode enables it to excite a brighter fluorescent signal than CW mode illumination with the same average laser power. Thus, TG mode imaging provides higher imaging contrast. To quality the imaging contrast, we calculated the intensity contrast using the following function:

    Icontrast=(ImaxImin)/Imax,

    where the Icontrast is the index of intensity contrast, Imax is the mean intensity of the fluorescent area, Imin is the mean intensity of the background area. We compare the Icontrast and intensity range of TG mode and CW mode with average laser power from 54 mW to 486 mW. Furthermore, we study the photothermal effect of these two modes. TG mode shows a lower photothermal effect than CW mode and the comparison results are shown in the supplementary information.

    To obtain the whole-body image of the vessel structure in tissue volume, we move the stage during light-sheet scanning, 100 μm a step, after capturing the image with an exposure time of 500 ms. It takes about 6.25 minutes to complete 750 steps for the whole mouse scanning. We then use Imaris software to reconstruct the 3D image.

    A conventional method to enhance the vessel structure is to calculate the partial derivative of the image by Hessian matrix25. However, when the vessels are presented against a complex background, the resolving power of Hessian matrix based method is limited. To enhance the vessel contrast and resolve the small vessels, we first make a compensation against the background gradient by rotating the profile to a horizontal angle and then mark the vessel by filtering pixels with the shape of uplifting. Although this approach can well judge the small blood vessels under a low contrast, it may misjudge the noise and the calculation speed is slow. To solve this problem, we introduce a deep learning model of DenseNet27. We modify the results of the above algorithm by denoising, and training the DenseNet model with raw images and the corresponding results. Finally, we obtain a depth neural network model which can improve the contrast between the blood vessel and background tissue. The 3D reconstruction result and its vessel enhancement result are shown in Fig. 5. The pixels in the white block are used to calculate the mean intensity of the background. There is a very weak vessel signal pointed out by the white arrow. This algorithm enables us to obtain the vessel structure from images with Icontrast down to 0.07 (−0.012+0.007)Icontrast, and to reveal more vessels when their intensities are close to the background. In a 55 mm × 75 mm × 20 mm volume, the finest size of the vessels was around 100 µm. Using light-sheet center location and vessel enhancement algorithm, we provide more depth information with more details about the vessel network than the traditional small animal imaging system. In Fig. 6, we reconstruct the 3D model of mice vessel. By measuring each length of the vessels, we conclude that the total length of the reconstructed vessel is 527.7 mm.

    Comparison schematics and results of vessel enhancement processes by using Hessian matrix v.s. DenseNet deep learning algorithm based on protuberance detection.(a) Raw image. (b) Hessian matrix enhancement. (c) DenseNet enhancement result. (d) Intensity discriminator. (e–h) are the profiles of white dotted lines in (a–d). (i) Training process of Hessian algorithm and our algorithm.

    Figure 5.Comparison schematics and results of vessel enhancement processes by using Hessian matrix v.s. DenseNet deep learning algorithm based on protuberance detection.(a) Raw image. (b) Hessian matrix enhancement. (c) DenseNet enhancement result. (d) Intensity discriminator. (eh) are the profiles of white dotted lines in (a–d). (i) Training process of Hessian algorithm and our algorithm.

    The depth information of the high contrast images of the whole mice skin and blood vessel revealed by the time-gated light-sheet NIR-II volumetric imaging system enhanced by deep learning algorithm. (a) The 3D reconstruction image of mice skin and vessel networks. (b) Luminescent intensity map. (c) The height map of the mice’s blood vessels. (d) The height map of the mice’s skin. (e) Vessels with colors that represent their depth under the skin. (f, g) Local details of the same position in (a) and (c). (h, i) Intensity profiles of vessels in different depths. (h) A magnified part of (b). Intensity profiles of lines in (i) with depths of 500 μm, 750 μm, and 1000 μm.

    Figure 6.The depth information of the high contrast images of the whole mice skin and blood vessel revealed by the time-gated light-sheet NIR-II volumetric imaging system enhanced by deep learning algorithm. (a) The 3D reconstruction image of mice skin and vessel networks. (b) Luminescent intensity map. (c) The height map of the mice’s blood vessels. (d) The height map of the mice’s skin. (e) Vessels with colors that represent their depth under the skin. (f, g) Local details of the same position in (a) and (c). (h, i) Intensity profiles of vessels in different depths. (h) A magnified part of (b). Intensity profiles of lines in (i) with depths of 500 μm, 750 μm, and 1000 μm.

    Moreover, to resolve the depth information of blood vessels from the mice skin, we designed a scattering-based modality to acquire the 3D surface profile of the mice skin. The height information of the mice vessel is shown in Fig. 6(c), and the mice skin is shown in Fig. 6(d). In our experiment, we used a 1550 nm laser to form a light sheet, as this wavelength is consistent with the emission wavelength of DSNPs to avoid chromatic error. Thus, the fluorescent imaging mode is to map the vessels and the scattering scanning mode is to image the skin profile, which can be integrated into the same system with the same FoV as shown in Fig. 6(a). To make sure the light sheet generating device is stable and consistent, both illumination optics share a single-mode fiber, and switching between the two modes can be realized by feeding the laser input from the other end of the fiber. By merging the two results of vessel image and skin surface image of the mice, the depth information of vessels under the skin surface is finally shown in Fig. 6(e). The longest depth of vessels we see from the skin is 1.95 mm. We analyzed three areas of the vessel part which show different depths under the skin. The red part and orange part are of similar width in the view of biology (they are symmetrical on the mice), however, the leaning gesture of the mice makes them have different distances to the skin from the side of view. More or less, can describe the relationship between fluorescence scattering and vascular depth.

    Conclusions

    We demonstrate that the synergistic use of the time-gated imaging, light-sheet scanning and deep learning algorithm can effectively minimize the influence of laser scattering and out-of-focus background noise, so that vasculature can be clearly resolved by NIR-II volumetric imaging through deep tissue. We show the volumetric imaging system can successfully reconstruct the total length of 527.7 mm of vessels from the whole mice thick tissue within 2 mm from the skin, including the smallest feature size of 0.1 mm diameter vessels. The whole mice scale 3D imaging with acceptable depth resolution of 0.1 mm fills the gap between the conventional 2D microscopic and macroscopic imaging, which enables the studies of the whole-body pathology models. With the large volume of information, the system provides capability of analyzing the microscopic details of macroscopic region of interest in the context of the intact body28, such as the transport of cancer cells in blood vessels29. High speed in vivo volumetric imaging with high contrast and high resolution offers superior advantages compared with the current tissue transparency techniques used to avoid scattering effects in imaging the details of fixed organs.

    There are several directions for further improvements. The emerging field of in vivo volumetric imaging will highly demand NIR cameras to be developed with high resolution and sensitivity. For example, the maximum number of pixels from the best available NIR-II cameras is currently limited at 640 × 512, which is far from being sufficient in resolving vessels with the smallest diameter towards only five micrometers30. Though the power of NIR-II imaging has been improved in imaging the gastrointestinal tract and bones of mice31, as the result of strong absorption and scattering by the tissue, the penetration depth for NIR-II illumination and emission detection from the NIR-II contrast agent through deep tissue remain limiting the optical system to resolve the vessel networks deeper than 2 mm from the skin. Our current design of the light-sheet scanning only allows one side of the mice to be illuminated, which leads to the mice skin partially obscured, and rotating scanning scheme can be used to cover the entire angles of the mice. Techniques, such as adaptive optics, multi-sheet illumination imaging, structured light illumination imaging, etc, can be adopted to enhance the performance. Last but not the least, the rapid progress in material sciences with bright and more efficient luminescent contrast agents to be developed will continue to accelerate both the fundamental research and pre-clinical translations of NIR-II volumetric imaging, as well as the field of imaging guided minimally invasive diagnostic and therapeutic procedures32.

    [4] [4] Du HT, Wan H, Dai HJ. Recent advances in development of NIR-II fluorescent agents. In Benayas A, Hemmer E, Hong GS, Jaque D. Near Infrared-Emitting Nanoparticles for Biomedical Applications 83–101 (Springer, 2020);http://doi.org/10.1007/978-3-030-32036-2_5.

    [27] [27] Iandola F, Moskewicz M, Karayev S, Girshick R, Darrell T et al. DenseNet: implementing efficient convnet descriptor pyramids. arXiv: 1404.1869 (2014).

    Tools

    Get Citation

    Copy Citation Text

    Sitong Wu, Zhichao Yang, Chenguang Ma, Xun Zhang, Chao Mi, Jiajia Zhou, Zhiyong Guo, Dayong Jin. Deep learning enhanced NIR-II volumetric imaging of whole mice vasculature[J]. Opto-Electronic Advances, 2023, 6(4): 220105

    Download Citation

    EndNote(RIS)BibTexPlain Text
    Save article for my favorites
    Paper Information

    Category: Research Articles

    Received: May. 31, 2022

    Accepted: Oct. 12, 2022

    Published Online: Jun. 13, 2023

    The Author Email:

    DOI:10.29026/oea.2023.220105

    Topics