To solve the problems of the low-accuracy detection or inaccurate classification of small target defects in solar cell panel defect detection, an improved lightweight YOLOv5s solar cell panel defect detection model suitable for small target detection is proposed in this study. First, an SiLU activation function is used to replace the original activation function to optimize the convergence speed and enhance the generalization ability of the model. Second, the C3TR and convolution block attention modules are used to re-optimize the backbone feature sampling structure to improve the recognition ability for different defect types, especially small target defects. Third, the content-aware re-assembly of features is realized in the feature extraction network to improve the detection accuracy and detection speed without increasing the model weight. Finally, a dynamic nonmonotonic loss function WIoUv3 is added to the dynamic matching prediction box and real frame to enhance the robustness of small target datasets and noise. Experimental results show that the mean average precision (mAP@0.5) of the proposed model is 95.9% and that its classification accuracies for large-area cracks and star-shaped scratches reach 98.0% and detection speed reaches 75.133 frame/s, demonstrating its lightweight nature and rapidness that meet the requirements of industrial production.