In order to address the shortcomings of the existing space error analysis methods of CNC machine tools, the geometry error compensation model was optimized based on Abbe principle. First, the geometry error compensation model of the three-axis machine tool was deduced, and the precondition for the correct usage of the model was provided. Second, the mechanism of spatial error transfer of three-axis machine tool was analyzed, based on the Abbe principle. The influence of Abbe error on the positioning accuracy of machine tool was determined, and the theoretical calculation formula was provided and verified by conducting experiments on the moving axis of machine tool. Finally, the existing geometric error compensation model of HTM was optimized based on the Abbe principle and Bryan principle.This model was used to fit the diagonal space error, and compared with the actual measured diagonal error of the machine tool. The existing HTM geometric compensation model can compensate for the machine tool space error, altering it from 41.15 μm to 16.37 μm, with a compensation rate of 60.22%. The optimized compensation model can compensate for the machine tool space error, altering it to 5.32 μm, with a compensation rate of 87.07%, which is an increase of 26.85%. The experimental results show that the optimized compensation model is more reasonable, and further improves the accuracy of space error compensation.