A defect detection model based on improved YOLOv7-tiny is proposed herein to address the problem of different detection methods for different defects on apple surface. Combined with RGB+NIR multispectral images collected by a camera, various defects on the apple surface are detected and classified. First, to extract more effective feature information and improve the ability to locate defects, coordinate attention (CA) is used to aggregate coordinate information in the backbone network, and a contextual transformer (CoT) module is added behind the backbone network to increase the global receptive field. Second, it is combined with the weighted bidirectional feature pyramid to adjust the proportion of each branch in the structure to enhance the feature fusion ability of efficient layer aggregation networks. Finally, the loss function is replaced by Focal-EIoU loss to solve the problem of unbalanced samples. The mean average precision (mAP) @0.5 of the improved network increases by 1.2 percentage points to 93.2%, and the recognition speed is 89.3 frames/s. The research content of this paper provides a more efficient method for apple surface defect detection and a more accurate basis for apple grading.