To improve the angle measurement accuracy of a circular grating encoder and meet the accuracy measurement requirements of arc-second and even sub-arc-second levels, this paper analyzes the error sources that affect the accuracy of the angle measurement and proposes the compensation method of the multi-reading head reading averaging method to control the error sources. According to the principle of the circular grating angle measurement, the error sources, such as the grating system engraving error, reading head interpolation error, grating installation eccentricity error, installation deformation error, and shaft shaking error, are analyzed, and each error spectrum is analyzed from the perspective of frequency domain. According to the error source analysis, the compensation method of multi-reading head reading averaging method can be used. Experimental results show that under the condition that the eccentricity error is about 15″ and the grating deformation error is about 1.5″, the measured angle error is better than 0.8″ by the four-reading head averaging method, which greatly improves the angular measurement accuracy of the circular grating.