Identifying the species of wild mushrooms is important to prevent mistaking the toxic type of mushrooms for non-toxic ones. Therefore, to improve the accuracy of the fine-grained classification of wild mushrooms, a parallel addition convolutional block attention module (PA_CBAM), which is improved from the convolutional block attention module (CBAM), is proposed. PA_CBAM changes the connections of the channel and spatial attention modules from serial to parallel and adds their results together. Consequently, the interference caused by cascading these attention modules is solved. In addition, the proposed method improves the performance of ResNet50 by referring to the concept of a feature pyramid, whose accuracies of the Top-1 and Top-5 are 86.03% and 97.19%, which are 0.86 and 0.73 percentage points higher than those of the original method, respectively. Furthermore, the Top-1 and Top-5 reach 88.52% and 97.58% using PA_CBAM, which are 3.03 and 0.69 percentage points higher, respectively. Moreover, to adapt the model for mobile terminals, combined with migration learning, the MobileNet_v2+PA_CBAM recognition method is proposed, obtaining an accuracy of 94.87%, which is 0.66 percentage points higher than that previously obtained. The results show that PA_CBAM has a better recognition and generalization effect in the fine-grained classification of wild mushrooms. Meanwhile, the size of MobileNet_v2+PA_CBAM is only 27.8 MB, and the recognition time required for a picture is only 1.3 ms, which is an ideal model for deploying wild mushrooms classification on mobile devices.