To solve the problem of difficulty in extracting features and poor real-time performance of existing photovoltaic power station defect identification methods, which lead to low identification accuracy of photovoltaic module defect detection, this paper proposes a photovoltaic power station infrared thermal imaging defect detection method based on an improved YOLO v5 algorithm. The improved YOLO v5 algorithm primarily adds an attention mechanism SE module to the original core and improves the loss function from GIoU to EIoU to enhance the model convergence effect. Finally, the knowledge graph (KG) module is used to balance the feature pyramid structure and optimize the model to improve the YOLO v5 algorithm’s recognition accuracy and convergence effects. The improved network structure was applied to the YOLO v5s model, whereby the average detection accuracy mAP used in the detection of infrared images of photovoltaic power plants reached 92.8%, which is 4.5% higher than that of the original YOLO v5s algorithm (88.3%). The effect of convergence on the precision and recall rate was also improved compared with the original YOLO v5 algorithm model. By applying the enhanced network structure to the three models (l, m, and x), detection accuracy was also improved. Consequently, the improved YOLO v5 algorithm is suitable for the four models.