To improve the quality of the spectral instruments, two ellipsoidal mirrors and an ellipsoidal aberration-reduced grating are used. Reflected by the first ellipsoidal mirror, the wave front of the light is changed, and after passing the grating the light is focused on different positions of the detector according to the difference of the wavelength by the second mirror. The spectral instruments are with complex aberration and imaging characteristics. According to Fermat principle, consulting the second generation varied-line-space grating's fabrication optical layout and theory, the coordinate system is setup, the angles of incidence and reflection are defined, the elements are located, and the basic geometrical relation is established. The optical path function is divided into two parts: one is associated with the diffraction, and the other is associated with the optical path change by the geometrical position. After the power series of the above two optical path are expanded, the analytical expressions of the high-order aberration are derived, which are useful in the plane, cylinder, spherical gratings, as well as reflector. And the expressions of the aberrations and the optical layout can be used in other spectral instruments.