The error of photoelectric encoder detection system is mainly affected by the angle measurement error of the reference photoelectric encoder, data acquisition error and coaxial error. The angle measurement error can be compensated. In this paper, an algorithm based on extreme gradient boosting (XGBoost) machine learning is designed to compensate the error of the reference photoelectric encoder. After compensation, the static accuracy is improved by 35 times. The standard deviation is decreased from 3.62" to 0.13", and the maximum error value is reduced from 5.53" to 0.39". Compared with the traditional back progagation (BP) neural network algorithm and radial basis function (RBF) neural network algorithm, XGBoost's compensation is better than the others. XGBoost machine learning algorithm compensation effectively reduces the measurement error of the reference photoelectric encoder and improves the detection accuracy of the photoelectric encoder detection system.