In view of the low efficiency and poor effect of the traditional mushroom feature extraction method, a lightweight mushroom image classification model is proposed. In view of the small dataset used in the experiment, this classification model initializes the EfficientNetV2 model and modifies the full connection layer in the migration study based on the Imagenet dataset. At the same time, in order to reduce the parameter influence in the network, the original EfficientNetV2 model is streamlined to remove duplicate modules in the network. Finally, the squeeze-and-excitation mechanism in the original MBConv module is replaced with the coordinate attention (CA) attention mechanism with better feature extraction effect, and the new CA-EfficientNetV2 network is obtained. The experimental results show that compared with the classical ResNet50 model and RegNet, the classification accuracy of the proposed EfficientNetV2 is improved by about 10 percentage points and 2 percentage points respectively, and higher generalization performance is obtained; compared with the original EfficientNetV2, the classification accuracy is improved by 3 percentage points. That is, CA-EfficientNetV2 has higher accuracy and classification performance in mushroom classification.