For the daily essential food of people, automatic damage detection and automatic classification are essential for the increasing consumption of fruit. In view of this demand, automatic detection of fruit damage has become a hot topic in recent years. In this paper, the application of convolutional neural network, an existing deep learning technology, in fruit feature extraction and classification was discussed. A method based on ResNet34 as the backbone network and the introduction of attention mechanism SE and CBAM module was proposed to realize the detection and basic classification of fruit damage. The method was verified on fruit fresh and rotten for classification data set, and compared with VGG16, GoogLeNet, MobileNetV2 and other common networks. The accuracy of fruit damage detection and classification is improved. The classification accuracy reaches 98.8%. By adding the new apple data set, the performance of the model is further improved, compared with the original network ResNet34, and the generalization of the model is effectively improved, which provides a reference for the complex multi-feature classification of actual fruit images.