This study was aimed at dealing with the reduction in the precision of the angle measurement system in precision reducers due to the installation tilt and the deformation of the circular grating. The traditional error compensation method is limited by the number of working faces of the multi-sided prism， making the compensation effect insufficient. Thus， an error compensation method based on multi-data correlation was proposed. The autocollimator was combined with a regular 24-sided prism and used to calibrate the angle measurement error of the circular grating. By changing the relative position of the polyhedral prism and the circular grating in the circumferential direction， several groups of calibration data for the measurement of the angle error were obtained. The harmonic wave error compensation method was then used to preprocess multiple groups of measurement data； subsequently， the different groups of data were correlated and trained by a back propagation neural network to obtain the angular error model for the circular grating. This method does not need to employ the compensation formula， and it can also overcome the problem of fitting oscillation data when there are several sample points in the harmonic wave error compensation method. The experimental results show that the peak value for the residual error of the proposed compensation method is within 0.94" and the standard deviation is 0.25"； these values are 33.3% and 37.9% less than those of the traditional harmonic wave error compensation method， respectively. The proposed method effectively improves the accuracy of the angle measuring system to a sub-arc second level.