Because of the effects of the background environment and data volume, the accuracy and efficiency of abnormal defects in traditional infrared images of insulators are generally low. In this study, a deep-learning anomaly diagnosis method is proposed. Based on the improved faster region-based convolutional neural network (R-CNN) method, a detection network is built to test different types of insulators. Results show that compared with the back propagation neural network and faster R-CNN methods, the proposed method can diagnose abnormal defects of insulators efficiently with a mean average precision of 90.2%. In addition, the diagnostic accuracy of single type I and type V insulators is higher than that of double type I insulators. The results can provide a reference for insulator defect identification in transmission lines.