Aiming at low efficiency of traditional visual detection method and shallow model as well as low semantic character of target detection algorithm based on manual features, an surface defect detection method of patch diode based on improved YOLO-V4 was proposed. Firstly, DenseNet was used in CSP1 module to replace ResNet in original network, considering that gradient disappeared with network deepening and feature redundancy as well as parameters were reduced. Then, to realize cross-dimensional interaction of feature information and make the network pay more attention to important information, the three-branch attention mechanism module was introduced after CSP1 module, and features were fused with FPN+PANet. CBL×5 module was replaced by CSP2, which reduced computation of network and improved detection speed of algorithm. Finally, the Focal Loss function was optimized and weight was added to positive and negative samples to solve the imbalance problems. The detection precision (P), recall ratio (R) and mean average precision (mAP) of the algorithm are 2.98%, 2.65% and 2.92% higher than that of YOLO-V4, respectively, which shows that the improved YOLO-V4 can effectively detect the surface defects of patch diode.