The developing trend and characteristics of the space optical system are long focal length, large view field, light weight, small dimension, large relative aperture and high image quality. The structural parameters of the coaxial three-mirror system can be solved and optimized by ZEMAX. The coaxial three-mirror system with focal length of 5 000 mm and F number of 10 is designed. The result indicates that the field of view of the system is 8°, and MTF is higher than 0.4 at 50 lp /mm. The image quality of the optical system approaches the diffraction limit, and can meet the different requirements of high-resolution space camera. This design has a theoretical guidance and significance for the design and selection of structure as well as manufacture of space camera IR optical system.