During the industrialized production of fabrics, fabric defects are varied and contain a large number of small target defects and elongated defects with extreme aspect ratios, which makes fabric defect detection a challenging task. To address this problem, an improved YOLOv5s algorithm is proposed in this paper. Firstly, the Mosaic data enhancement method is improved, which enriches the dataset while weakening the side effects of the original data enhancement method on the detection of some fabric defect types, and improves the detection of small targets and extreme aspect ratio defects. Then, the batch normalization is improved to representative batch normalization, which improves the algorithm’s differential feature representation of diverse defect instances and suppresses noise interference; finally, the lightweight coordinate attention is introduced, which encodes the long-distance dependency and channel dependency of the features with accurate location information, and enhances the algorithm’s ability to locate defects. The experimental results show that the algorithm in this paper significantly improves the detection ability of small targets and extreme aspect ratio defects, making the average detection accuracy mAP reach 81.3, which is 4.1% higher than the original YOLOv5s, and the detection speed is 32.6 fps, which fully meets the real-time requirements, and the algorithm better balances the detection accuracy and detection speed.