Breast cancer is among the most common malignant tumors and a serious threat to women’s health. Therefore, rapid and efficient screening for breast cancer is increasingly important. Currently, imaging and pathological examinations are the two main methods used for breast cancer diagnosis. Imaging examinations typically have shortcomings such as long examination time and radioactivity. Pathological examination, which is the gold standard for cancer diagnosis, has the disadvantages of complicated preparation and time-consuming processes. Therefore, a new intelligent method must be developed for breast cancer diagnosis to reduce the reliance on traditional techniques. Microscopic imaging and fluorescence spectroscopy are crucial tools for studying the characteristics of cancerous breast tissues and for effectively capturing changes in tissue morphology and biochemical composition. In this study, a self-designed and processed inverted fluorescence microscope was employed to simultaneously collect microscopic images and fluorescence spectra from breast tissues of patients.

First, the samples used in this study were obtained from patients with invasive breast cancer. The fresh tissue samples were cut into tissue blocks of approximately 3 mm×3 mm×2 mm size, quickly frozen within liquid nitrogen, and then cut into tissue sections with a thickness of 15 μm. The entire process did not require staining of tissue sections. Next, microscopic images and spectra were collected. The collection equipment was designed and customized based on an inverted fluorescence microscope (IX51, Olympus). Bright field and fluorescence imaging modes of different wavelengths were achieved by switching excitation filters and dichroic mirrors and adjusting the light source. A total of 69 sets of multimodal microscopic images and 46 sets of spectral data (divided into purple and blue light excitation) were obtained from the tissue sections of 23 patients. The spectral data were preprocessed using baseline correction and third-order polynomial 30-point Savitzky?Golay smoothing. Finally, a neural network model was constructed based on multimodal microscopic images and fluorescence spectra that included image feature extraction, spectral feature extraction, and spectral feature fusion. For image feature extraction, multimodal images were stacked into a three-dimensional (3D) matrix, and joint feature extraction was performed with the help of 3D convolutional layers and residual modules. For spectral feature extraction, the fluorescence spectra excited by excitation light of different wavelengths (purple and blue) were first stacked in parallel into a two-dimensional (2D) matrix, and joint feature extraction was achieved with the help of a 2D convolution layer and residual module. The extracted image and spectral features were then combined, and the spectrum?image fusion features were further explored with the help of a fully connected neural network, ultimately achieving an intelligent diagnosis of breast cancer.

The first step is a multimodal microscopic image analysis. Under purple light excitation, bright field images, blue fluorescence images, and their fusion images show that the extracellular matrix of normal breast tissue has a more uniform fluorescence distribution than that of cancerous breast tissue (Fig. 3). Under blue light excitation, bright field images, green fluorescence images, and their fusion images show that the cancerous breast tissue has a strong green fluorescence signal (Fig. 4). The fluorescence spectrum analysis shows that the average spectral intensity of normal breast tissue under purple light excitation is significantly stronger than that of cancerous breast tissue, and the fluorescence intensity of cancerous breast tissue under blue light excitation is slightly higher than that of normal tissue (Fig. 5). A Gaussian function was used to fit and analyze the spectra excited by purple light. The analysis results show that the area ratios (A₅₂₀/A₄₇₀ and A₆₃₅/A₄₇₀) of the fluorescence spectrum peaks at the central wavelengths of 520 nm and 635 nm to that at 470 nm in cancerous breast tissues increase by 0.65 and 1.07 times, respectively, compared to those of normal breast tissues (Table 2). Finally, during intelligent diagnostic analysis, the training and prediction results of the spectrum?image fusion neural network show that the risk of model over-fitting is extremely low. The calculated area under the curve (AUC) is 0.95, indicating that the model has good classification performance. Further calculations of the test set data show that the average accuracy is 86.38% (Fig. 7). At the same time, this study further lists the training results of five types of image and spectral data used alone for breast cancer diagnosis for comparison with the diagnostic results of spectrum?image fusion data. The results indicate that the spectrum?image fusion neural network can achieve a significantly higher prediction accuracy than each single-modality model (Table 3).

In this study, dual-modal microscopic imaging and multiwavelength microfluorescence spectroscopy are combined to diagnose breast cancer to obtain more comprehensive information on compositional changes in breast tissue during canceration. The Gaussian fitting model is used to perform peak splitting analysis on the spectral data. Changes in endogenous fluorophore content during cancerization are discussed, and the peak area ratio is proposed as a potential criterion to diagnose breast cancer. In addition, this study introduces deep learning to construct a spectrum?image fusion neural network model based on fluorescence microscopic images and microscopic spectra, achieving an AUC score of 0.95 and an accuracy of 86.38% that are significantly higher than those of each single-modality model. This provides a feasible method for the intelligent diagnosis of breast cancer with the advantages of convenience, speed, and clinical significance.