As a high-precision non-contact measurement method, spectral confocal displacement measurement technology is widely applied in high-precision fields such as semiconductor manufacturing, precision machining, and material science due to its excellent performance in micro-nano displacement and morphology measurements. However, in actual processing, the sample surface is often not an ideally smooth plane. Instead, it is characterized by varying degrees of micro-fluctuations and surface roughness. A complex scattering phenomenon is caused by surface roughness when incident light interacts with the sample surface. The characteristics of the reflected light beam are altered by this scattering, which in turn leads to the peak wavelength positioning of the spectral signals being affected. As a result, measurement errors are introduced, which lead to a reduction in the accuracy of the displacement measurements. Especially in the high-precision measurement scene, this situation has been recognized as an important bottleneck restricting the improvement of system performance. To enhance the applicability and measurement accuracy of spectral confocal displacement measurement systems, the influence of surface roughness on displacement measurements is systematically analyzed, and an innovative error correction algorithm model is proposed in this paper to address this challenge. Firstly, the working principle of spectral confocal displacement measurement system is introduced. Then, based on the light scattering theory, the small slope approximation (SSA) method is used to analyze the theory. The SSA method can be utilized to analyze the variations in reflectivity characteristics resulting from the interaction between a light beam and a rough surface. Additionally, it can be employed to establish a mathematical relationship model between surface roughness and displacement measurement error. The theoretical foundation is provided by this model for understanding the influence of surface roughness on spectral confocal measurement. Subsequently, the displacement measurement error caused by the drift of the peak wavelength curve due to surface roughness is comprehensively analyzed through theoretical investigation and simulation. The simulation results indicate that the measurement accuracy of the spectral confocal system is reduced by surface roughness. Specifically, with the increase of surface roughness, the deviation of axial displacement becomes more significant. To address this issue, an error correction method combining Standard Normal Variate (SNV) transformation and machine learning techniques is proposed in this paper. Firstly, the SNV transformation is applied to preprocess the spectral data, and a roughness error correction model is established to improve the accuracy of spectral peak identification. To further refine the correction process, the Support Vector Machine (SVM) algorithm is employed to enhance accuracy. The SVM model is trained on spectral data corresponding to different roughness levels to correct the displacement measurement error caused by surface roughness. Carbon steel samples with controlled surface roughness values of 10 nm, 15 nm, 20 nm, 25 nm, 30 nm, and 35 nm are used as test objects for experimental validation. The experimental results validate the effectiveness of the modified model: the relative standard deviation of displacement measurement values is reduced from 1.59% to 0.12% following the application of the SNV-SVM correction method. The significant reduction in measurement deviation demonstrates that the correction model proposed in this study effectively mitigates the influence of surface roughness on spectral confocal displacement measurement. The research results presented in this paper provide a theoretical and practical basis for enhancing the measurement accuracy and applicability of the spectral confocal system. By addressing the challenge of surface roughness, high accuracy is maintained by the system, even when samples with varying roughness levels are measured, through the proposed calibration algorithm. In most cases, the research results presented in this paper hold significant guiding value for improving the measurement accuracy of the system and its applicability to a range of different samples.