As geometric errors of rotary axes of a five-axis machine impact on its machining accuracy, the on-machine measurement of rotary axes and their error modeling were investigated. Firstly, the measurement method for comprehensive errors of rotary axes was presented based on a standard ball and an on-machine measurement system. The measurement angular positions of the rotation axes were planned by the random Hammersely sequence and the initial position of the standard ball was determined with a free installation strategy. Then, a Radial Basis Function (RBF) neural network model for predicting comprehensive errors of the rotary axes was built based on subtractive clustering and Fuzzy C-means(FCM) cluster. Finally, mathematical analysis was carried out to provide a new way to model accurately for geometric errors of rotary axes. A cambered measuring example was used to verify the proposed on-machine measurement method and modeling method.The experimental results indicate that the average deviation between the predicted points from the mathematics model and measured points is 9.6 μm and the maximum deviation is not more than 15 μm. After the measuring results are corrected by the mathematic model with a 3D coordinate measuring machine established by this paper, the average value is reduced to 13.6 μm from 33.5 μm and the maximum value is reduced to 18.6 μm from 62.3 μm. It concludes that the measurement method is simple to operate and has high automation. The neural network's training is not only fast speed, adaptable and good robust, but also can meet highly linear and strongly coupling of modeling of the geometric errors of rotary axes.