Cashmere is characterized by lightness and comfort, smoothness and softness, dilution and breathability, and good warmth. Because it is very expensive, the quality of cashmere products in the market is mixed. Existing microscopy, DNA, chemical dissolution, and image-based methods have shortcomings such as damaged samples, expensive equipment, and high subjectivity. NIR spectroscopy is a rapid measurement method that is non-destructive and allows for modeling operations. Aiming at the problems that traditional modeling methods usually fail to learn universal near-infrared spectral band features, resulting in weak generalization ability, and that the near-infrared spectral band features of cashmere wool fibers are similar and difficult to distinguish, this paper proposes a near-infrared spectroscopy cashmere wool fiber prediction model based on two-way multi-scale convolution. In terms of data preparation, a total of 1 170 near-infrared spectral band data of the original cashmere wool samples are collected for validation, and the range of the near-infrared spectral band data is 1 300~2 500 nm; in terms of model design, two parallel convolutional neural networks are utilized to extract the features of the near-infrared spectral band, and both the original near-infrared spectral band data and the downscaled near-infrared spectral band data are used as simultaneous. The original near-infrared spectral band data and the downscaled near-infrared spectral band data are input simultaneously. The intermediate contributing near-infrared spectral band features are further extracted using the multi-scale feature extraction module, and the path exchange module is used for the information exchange of the two near-infrared spectral band features. Finally, the cashmere wool fiber prediction results are obtained using the class-level fusion. In the experimental process, 80% of the collected near-infrared spectral band data are used for model training and 20% of the near-infrared spectral band data are used for model testing. The average prediction accuracy of the test set of the model proposed in this paper is 94.45%, which is improved by 7.33%, 5.22%, and 2.96%， respectively, compared with the traditional algorithms such as Random Forest, SVM, and 1D-CNN, etc. Ablation experiments are conducted to further validate the structure of the proposed model. The experimental results show that the proposed two-way multi-scale convolutional near-infrared spectroscopy cashmere wool fiber prediction model can realize the fast and nondestructive prediction of cashmere wool fibers, which provides a new idea for the prediction of cashmere wool fibers in near-infrared spectroscopy.