Spectral confocal displacement sensor is a new geometric precision measurement sensor with high accuracy, high efficiency, and non-contact technical advantages. It is now widely used in measuring micro or macro geometric quantities. Conventional geometric measurement sensors use contact mechanical probes, which cause damage to the surface of the object, making it difficult to meet the needs of non-destructive measurement in modern manufacturing. Unlike conventional optical systems that require correction for axial chromatic aberration, spectral confocal displacement sensors use axial chromatic aberration to establish the relationship between displacement and wavelength. However, most current research on spectral confocal displacement sensing technology has focused on point sweep. This technique can only obtain the geometric information of a single point, which greatly limits the efficiency in the practical application of precision measurement of larger areas and requires high back-end data processing and cumbersome data reconstruction. This study designs a line-swept spectral confocal displacement sensor system with submicron resolution to address this technical drawback. This study analyzes the principle of the line-swept spectral confocal displacement sensor and the detailed design of a large-range dispersive objective lens and a high spectral resolution spectral spectroscopic unit is carried out. By optimizing the optical path structure of the dispersive objective and spectral spectroscopy unit and balancing the aberrations, the RMS radius of each field of view of the full system is less than 5.5 μm, and good imaging quality is obtained. The results show that the full system has a resolution of 0.8 μm at a scan line length of 10 mm and an axial range of 3mm. This study has a broad application prospect in high-efficiency and high-precision geometric precision measurement.