Glioma is an invasive malignant primary brain tumor, characterized by its infiltrative growth pattern, making it challenging to differentiate its boundaries from healthy brain tissues during surgical procedures. Our study aims to address the challenge of accurately distinguishing glioma from healthy brain tissues. Based on a home-made high-resolution polarization-sensitive optical coherence tomography (PS-OCT) system, we conduct ex vivo imaging of normal mouse brain and human glioma model mouse brain. Further, based on the polarization analysis of the tumor and normal brain tissues, we propose a novel tumor differentiation metric called optic axis standard deviation to distinguish normal and glioma tissues. It is proven by the results that high-resolution PS-OCT has great potential in imaging brain tissue and can offer enhanced precision to detect glioma during surgical interventions, which will help to solve the urgent clinical need in neurosurgical practice.

A home-made spectral domain PS-OCT system is constructed by utilizing a superluminescent diode as a broad-spectrum light source. The system is performed with an axial resolution of 3.4 μm in air and a transverse resolution of 4 μm in the focal plane. The low-coherence light beam emitted by the broad-spectrum light source is linearly polarized in the vertical direction after passing through a linear polarizer. Through the integration of a quarter-wave plate and a polarizing beam splitter, circularly polarized light enters the sample arm, while orthogonally polarized signals are detected in the detection arm. After collecting the spectral signals corresponding to the two orthogonal polarization states, the parameters of intensity, phase retardation, and optic axis are calculated and the corresponding OCT images are obtained.

Given the similarity of genomes between the mouse and human, the mouse brain is chosen as the imaging object in this study. A human glioma mouse model is established by injecting U87-GFP human glioma cells into the mouse brain. The model is euthanized and fixed 5 weeks after establishment. Subsequently, green fluorescent protein (GFP) fluorescence imaging and PS-OCT imaging are performed on the mouse brain. Three-dimensional field of view of PS-OCT imaging is 6 mm×3 mm×2.3 mm. The three-dimensional image contains 1000 B-scans, and each B-scan image consists of 2000 pixel×2000 pixel (x and z directions). The image in the x-y direction is extracted from the three-dimensional image to obtain the en-face image along the cross section of the sample.

In the OCT images of the normal mouse brain (Figs. 2 and 3), neural fiber bundles in different directions can be clearly observed. In the polarization images, these fiber structures are more pronounced, manifested as radiating short lines. The mouse cerebral cortex exhibits polarization-maintaining properties without birefringence changes. The internal capsule structure between the cortex and internal brain regions is visible with birefringent properties, leading to notable variations in phase retardation. The hippocampal structure lacks birefringent characteristics, resulting in a low value of the phase retardation.

Contrasting with fluorescence imaging results to determine the tumor location in the mouse brain (Figs. 4 and 5), it can be seen from the OCT images that there is no significant birefringence change in the tumor tissue, the color of the polarized images becomes more homogenous, and we can find that the infiltration of tumor cells leads to strong fiber damage, and there is no prominent linear structure in the optic axis results.

The optic axis histogram (Fig. 6) illustrates distinctive distribution features between normal brain tissue and glioma tissue. The optic axis of normal brain tissue is uniformly distributed between -π/2 rad and π/2 rad, while the optic axis in the glioma region is mainly concentrated near 0 rad. By calculating the standard deviation of the optic axis value within the spatial window, the evaluation parameters of optical axis standard deviation are established. The range and mean value of optic axis standard deviation in normal brain tissue far exceed those in glioma tissue. The quantitative analysis underscores substantial distinctions of polarization information between tumor and normal mouse brain tissues.

Within brain tissue, myelinated nerve fibers exhibit pronounced scattering and birefringence properties. The results of this study indicate that normal mouse brain tissue, rich in myelinated nerve fibers, is effectively identified by PS-OCT. The high-resolution intensity map contains detailed information, and the phase retardation highlights the birefringence changes of structures such as internal capsules and nerve fiber bundles. The optic axis values reflect the oriented arrangement of fibers, collagen, and other structures within the tissue. In the regions infiltrated by gliomas, the original structure of brain tissue is destroyed, the invading cancer cells decompose and reduce the expression of myelin, and the birefringence changes are weak. Compared with intensity images, polarization images exhibit higher contrast, providing a direct display of the distinctions between gliomas and normal tissue, making it easier to diagnose in clinical settings.

Our preliminary study only used a limited number of mice. More samples and glioma tissues at different stages will be further studied in the future study. In addition, efforts will be made to investigate in vivo animal samples and ex vivo human specimens, aiming to promote the application of PS-OCT in the identification and resection of glioma in neurosurgery of the brain.

Our results demonstrated that en-face images reveal structural information in the mouse brain, including the cerebral cortex, internal capsule, and hippocampus. Using polarization parameters of phase retardation and optic axis, the position and orientation of internal structures like the internal capsule and nerve fiber bundles with birefringent tissues can be clearly observed. The originally symmetric structure in the brain of tumor mice is destroyed, with erosion of the cortical edge and only partial visibility of internal fiber structures. Optic axis standard deviation proves effective in distinguishing between normal and glioma mouse brains. This study proves that high-resolution PS-OCT is promising to monitor the infiltration process of glioma through changes in birefringent tissues such as nerve fiber bundles.