To solve the problem of low classification accuracy of hyperspectral image classification method based on deep learning for small-sized samples, a classification model based on multi-scale residual network is proposed. By adding a branch structure into the residual module, the model constructs extraction modules based on spectral features and spatial features, respectively, realizes the multi-scale extraction and fusion of spatial and spectral features, and makes full use of the rich spatial and spectral information in hyperspectral images. In addition, dynamic learning rate, batch normalization, and Dropout are used in the proposed model to improve computation efficiency and prevent overfitting. Experimental results show that the proposed method achieves 99.07% and 99.96% of the overall classification accuracy on the datasets of Indian Pines and Pavia University, respectively. Compared with support vector machines and existing deep learning methods, the proposed model effectively improves the classification performance of small-sized sample hyperspectral images.