In order to overcome the shortcomings of the existing direct design method of freeform surface and the large amount of calculation, according to the structural characteristics of the off-axis display system, a three-dimensional direct design method of freeform surface based on Marius's law is proposed. First, according to Marius's law, all the characteristic data points of the freeform surface are solved in the full field of view and the full aperture. Then the characteristic data points are fitted into a freeform surface characterized by polynomials, and the initial structure of freeform off-axis system with good imaging quality is obtained directly. Finally, the polynomial coefficient is optimized with the merit function and the best fitting coefficient is determined. The required freeform surface mirror is generated and the final structure of the off-axis head-mounted display optical system is obtained. This method simplifies the design process and has high calculation efficiency. Based on the proposed method, a single-mirror head-mounted display optical system and a two-mirror head-mounted display optical system are designed with an entrance pupil diameter of 3 mm, a field angle of 19.12°×14.4° and an entrance pupil diameter of 8 mm, a field angle of 23°×16°, respectively. The design results show that single/two-mirror head-mounted display system designed by proposed method has good imaging quality and compact system structure. Tolerance analysis shows that the single/two head-mounted display system can finally achieve MTF greater than 0.3 lp/mm and 0.35 lp/mm in full field of view after introducing tolerances.