Deep learning technology is proposed to solve the social problem of the frequent occurrences of wild mushroom poisoning in China. However, due to the small difference between classes and complex image backgrounds, fine-grained recognition accuracy is low. To solve this problem, this paper proposes an improved ResNeXt50 network. First, a multiscale feature guide (MSFG) module is designed, which guides the network to learn and use low and high-level features fully through short connections. Then, the improved attention mechanism module is used to reduce the network’s learning for complex backgrounds. Finally, the different hierarchical features in the model are fused, and the obtained joint features are used for recognition. Experimental results show that the accuracy of the proposed network on the test set can reach 96.47%, which is 2.64 percentage points higher than the unimproved ResNeXt50 network. Comparison results show that the accuracy of the improved network model is 8.10 percentage points, 5.13 percentage points, 3.24 percentage points, 3.30 percentage points, and 4.25 percentage points better than VGG19, DenseNet121, Inception_v3, ResNet50, and ShuffleNet_v2, respectively.