Aiming at the problems of complex and inaccurate classification of benign and malignant brain tumors, a classification model was proposed based on the fusion of multi-scale and channel features. The model used ResNeXt as the backbone network. First, the multi-scale feature extraction module based on dilated convolution was used to replace the first convolution layer, which can make full use of dilation rates to obtain the image information from different receptive fields, and combine the global features with significant subtle ones. Second, the channel attention mechanism module was added in the network to fuse the feature channel information in order to increase the attention to the tumor, and reduce the attention to redundant information. Finally, the combination optimization strategy, the MultiStepLR strategy of the learning rate, the label smoothing strategy and the transfer learning strategy on medical images were adopted to improve the learning and generalization abilities of the model. The experiments were carried out on BraTS2017 Dataset and BraTS2019 Dataset, and the classification accuracy were 98.11% and 98.72%, respectively. Compared with other advanced methods and classical models, the proposed classification model can effectively reduce the complexity of the classification process and improve the detection accuracy of benign and malignant brain tumors.