Monochromators are widely used in spectral calibration, material analysis and other aspects, so research of high spectral resolution monochromator systems is of great significance. Based on the vector grating equation, the influence of the height of the incident slit on the spectral line bending of a spectrometer is investigated, and the analytical expressions of the spectral line bending at the same wavelength and the slit height are given. An optimization scheme of the spectral resolution of the monochromator based on the suppression of spectral line bending by the slit height is proposed. According to the performance index requirements of a highly sensitive and ultra-fast time response detector, a three-grating monochromator optical system with a spectral resolution of 0.1 nm and a band range of 185-900 nm was designed, and a prototype was built to verify the influence of the slit height on spectral line bending, and to explore the influence of slit height on spectral resolution on the above basis. The experimental results show that the spectral resolution can be improved from 0.32 nm to 0.1 nm by optimizing the slit height when the slit width is fixed.