A novel DeepLabv3++ model is proposed to address the low accuracy in identifying small targets and slow detection speed in fabric defect detection tasks. Firstly, a multi-scale lightweight backbone network is designed to extract features from defects with various shapes and sizes. Secondly, convolutional attention modules and channel spatial attention modules are introduced to capture boundary information of small targets and focus on defect regions. Additionally, two types of multi-level feature fusion (MFF) modules are added to mitigate the issue of detail information loss in decoder. Finally, the model is trained and evaluated using a fabric defect dataset collected from an industrial site. The results show that our DeepLabv3++ model outperforms other models, utilizing only 4.1 million parameters. It achieves a mean intersection over union (mIoU) of 90.01% and a mean pixel accuracy (mPA) of 95.05%, meeting the industrial site requirements for balancing detection precision and processing speed.