A fabric surface defect classification technique based on spatial attention multiscale feature fusion is designed to address the problem of low-classification accuracy caused by complex texture and varied defect kinds of the fabric surface. The multiscale pyramid pooling module is used to maintain the information integrity of the feature map, and the rich semantic information extracted from the high-level feature map is used as a priori information to guide the low-level features, realizing the fusion of high-level and low-level features; the improved spatial attention module is integrated into a convolutional neural network to enhance the differential expression of features. The improved class activation mapping method is used to obtain the defect classification information and location information. The fabric surface defect image is recognized and detected using data augmentation and transfer learning methods. The experimental results show that the proposed algorithm can effectively increase the accuracy of fabric defect classification and obtain defect location information without manual location labeling.