The surface defects of optical elements, namely surface defects, will directly affect the performance of the optical system. In the classification of surface defects, the shapes of many surface defects are irregular, so it is difficult to achieve the expected effect by relying on normal pattern recognition technology. To overcome the low precision and long time consuming in classification of surface defects of precision optical elements, a classification method of surface defects based on convolutional neural network was proposed. Firstly, the surface defect image was obtained by scattering method to analyze its imaging characteristics, and the training ability of the network was strengthened by rotating the image and mirroring the amplified dataset. Furthermore, the AC training network model was used to strengthen the feature acquisition ability of the network without increasing the extra calculation. Finally, the Softmax classifier was used to classify the surface defects into scratch, pitting and noise. The experimental results show that the defect classification accuracy of the used model is more than 99.05%.