A convolutional neural network （CNN） can be used in the industrial production environment to identify and classify textile defects. To overcome the problems in the visual discrimination of small defect types and imbalance of textile defect categories in actual scenes， a textile defect recognition network （TDRNet） based on label embedding method is proposed. First， the backbone structure is adjusted to improve the classification accuracy of the model. Then， a label embedded module （LEM） is constructed to generate the category weight offset of the model. Subsequently， a distribution perception loss function （DP loss） is proposed to adjust the class distribution of the algorithm； this reduces the distance of homogenous defect features and increases the distance of heterogeneous features. Finally， the seesaw loss function is introduced to dynamically balance the gradient update for different samples during the model training process by suppressing the negative sample gradient of a few categories and increasing the sample loss during misclassification， thereby alleviating the misclassification rate of a few categories. In the self-made "Guangdong intelligent manufacturing" cloth defect classification dataset， the top1 error rate of our framework for rough-grained and fine-grained classifications reached 16.35% and 17.12%， respectively， whereas the top5 error rate of fine-grained classification was as low as 5.20%. Compared with other classification models， TDRNet achieved the best results. In addition， TDRNet was compared with the classical fine-grained classification model in recent five years and achieved state-of-the-art （SOTA） performance， fully demonstrating the enhancements provided.