An improved YOLOv5s network for defects detection for the cable surface of cable-stayed bridge fast and accurately is proposed. This overcomes the problems of low efficiency and poor safety of manual inspection, slow and inaccuracy of existing target detection methods because of the interference of dirt leading to wrong and missed detections. The TRANS module is added to the backbone network of conventional YOLOv5s to obtain more features of a single image and improve defect detection accuracy. Moreover, the CSP module of the neck network is replaced by the GhostBottleneck module and ordinary convolution is replaced by depth-separable convolution to reduce parameters and improve the computational speed of the network. Furthermore, the SIOU loss function is used for suppressing the oscillation of the bounding box and improving the calculation accuracy of repeatability between the prediction and the real box, which can increase the model stability. The experiments show that mAP and FPS of improved YOLOv5s network are 94.26% and 68 frames per second, respectively. The performance is better than that of Faster-RCNN, YOLOv4, and conventional YOLOv5, and it can find the surface defect for the cable of the cable-stayed bridge accurately and timely.