Based on the idea of factorization neural network and residual structure, a convolutional block attention module for residual factorized of convolutional neural networks (CBAM-RFNet) is proposed by expansive convolution and adding attention mechanism. In this network, the traditional 3×3 two-dimensional convolution is decomposed into two one-dimensional convolution of 3×1 and 1×3 and connect them in series, which not only increases the depth of the network model, but also reduces the parameters, the network is a lightweight network model. The experimental results on thyroid cancer images collected by microhyperspectral imaging system show that, compared with other deep neural networks, the proposed network can effectively improve the classification accuracy of microhyperspectral images, with the overall accuracy of 98.23%, F₁ value of 98.66%, and Kappa coefficient of 0.909.