Cancer research has increasingly focused on developing appropriate tumor cell invasion models and developing a quantitative method to monitor tumor cell invasion. In this study, a three-dimensional tumor invasive model with >1 mm thickness is constructed. An ultra-wideband spectral domain optical coherence tomography system is used to detect cell migration and invasion dynamics, and the in vitro invasion process of tumor cells is characterized by the change of cell migration distance and matrix material decomposition. The quantitative detection of tumor cell migration distance based on peak change of the optical coherence tomography scattering interface is combined with three-dimensional images to quantify the matrix surface curvature, thickness, and overall volume change, thereby realizing the characterization of the matrix material decomposition and deformation information during tumor cell invasion. The changes of cell cluster positions caused by tumor cell invasion and morphological changes of matrix materials are matched with hematoxylin-eosin staining sections and laser confocal results, which verifies the feasibility of optical coherence tomography for detecting tumor-cell invasion. Three-dimensional tumor models under different nutrient gradients and different pH microenvironments are utilized. The established optical coherence tomography system accurately quantifies the migration distance of tumor cells and surface curvature and overall volume change of matrix materials at different time and in vitro microenvironments. Compared with hematoxylin-eosin staining and the laser confocal imaging method, the proposed optical coherence tomography-based method enables the continuous monitoring of the invasion process of tumor cells, thereby providing a more comprehensive view of tumor-cell migration and invasion mechanisms.