In order to enhance the optical efficiency, splitting prisms and holographic gratings are used to expand the exit pupil. And double-grating design is used to enhance the image quality. By modeling the system with optical design software, the optical quality is analyzed and simulated. The system has a field of view (FOV) of 20°×20°and an exit pupil of 20 mm×20 mm. The residual wave aberration is less than 0.075λ, the distortion is less than 0.01%, and the modulation transfer function close to the diffraction limit. For the zero FOV, the exit pupil has uniform irradiance distribution with an optical efficiency of 5%~6%. For the FOV of 20°, the irradiance uniformity drops down slightly, and the optical efficiency is 3%~4%. The simulated results show that the holographic waveguide system dramatically improves the optical efficiency of the holographic waveguide head-mounted display (HMD) system. Also the prisms waveguide optical system is highly practical and can be applied to the new generation HMD technology.