Photovoltaic (PV) panels are an important component of photovoltaic power stations. They must be tested regularly to ensure a safe operation of the photovoltaic power station. To address the problem of small targets being difficult to detect among the complex background of aerial photovoltaic images, a defect detection method based on YOLOv7-EPAN for infrared photovoltaic panel images is proposed. First, an extended efficient network CS-ELAN module integrated with a CSWin Transformer is proposed for capturing global information effectively and suppressing background information. Second, an efficient path aggregation characteristic pyramid network (EPAN) is constructed based on CS-ELAN to enhance the information interaction between different feature layers, enrich the semantic feature information, and improve the feature expression ability. Finally, the loss function is optimized to focus the model on a prior high-quality frame and improve the positioning accuracy of small targets. The experimental results show that compared with the original YOLOv7 model, the mAP₅₀ and mAP_50:95 of the proposed method show an improvement of 6.4% and 3.3%, respectively, indicating that the proposed method can better solve the problem of missing small target defects among the complex background of aerial photovoltaic images.