The XUV optical system composed of mirrors and gratings has the property of plane- symmetry. Combining the wave aberration theory of the Lu plane symmetric grating system and the root mean aquare aberration evaluation function, the multi-parameter objective function for optimizing the XUV optical system is obtained. In order to solve the objective function with multiple independent variables and a large value interval, a real-coded genetic algorithm based on decimal is developed in this paper, and it is used for the optimization of two XUV optical systems. The optimized value obtained is traced with the optical simulation software Shadow, and compared with the reference. The results show that the optical system optimized in this paper has a significant improvement in imaging quality, indicating that the wave aberration theory and the real-coded genetic algorithm in this paper are effective in optimizing the XUV optical system, which provides a new idea for the optimal design of such systems.