The manual classification methods for surface defects of precision optical elements are inefficient and the accuracy is easily affected by manual factors such as fatigue. And the accuracy based on traditional machine learning methods needs to be further improved. We propose an inspection method for surface defects of large-caliber optical elements using deep learning convolutional neural network. Firstly, collect and catalog a dataset of the surface defects of large-caliber optical elements through field tests. Then, for mining deeper feature expression, creat gradient-based three channels fusion image by the single-channel grayscale image. Finally, put forward the ICFNet which aims at Inertial Confinement Fusion (ICF) based on typical LeNet. The ICFNet does not require sophisticated manual design or feature extraction, only uses grayscale image to realize efficient inspection for surface flaws of large-caliber optical elements. Experiments show that ICFNet has better classification accuracy than traditional methods using multiple features and support, vector machines for three types of defects, including scratch, dust, and pits. This method has certain application value in surface defects identification of optical components.