The Taiji program is a key task for China's space gravitational wave detection, and as an important part of space gravitational wave detection, the telescope's performance will directly affect the accuracy of gravitational wave detection. For the existing typical space gravitational wave telescope structures, due to the high sensitivity of secondary mirror, it is difficult to meet the requirements for manufacturing and adjustment tolerance of larger aperture space gravitational wave telescopes, especially the tolerance requirements for in-orbit stability. In order to solve the above problems, firstly, a new optical system structure of space gravitational wave telescope with intermediate image plane set between three and four mirrors is proposed to reduce the sensitivity of the secondary mirror. Combined with the theoretical method of Gaussian optics, the initial parameters of the structure of the new telescope are theoretically analyzed and calculated. Secondly, through the optimization design, a telescope optical system with a pupil diameter of 400 mm, a magnification of 80 times, a field of view of ± 8 μrad, and a wavefront error RMS value of better than 0.0063λ was obtained. Finally, the sensitivity evaluation tolerance allocation table of the telescope system is established, and the tolerances of the existing telescope structure and the new telescope structure are compared and analyzed. Compared with the existing telescope structure, the sensitivity of the new telescope structure is reduced by 30.4%. The results show that the new telescope structure has the advantage of low sensitivity, which provides an optimal scheme for the design of space gravitational wave telescopes.