Remote sensing image classification is a key branch in remote sensing image processing, which provides an important basis for agricultural, industrial, and military applications. With the development of remote sensing satellite, spectral imaging technique has also developed from multi-spectral technology to hyperspectral technology. Rich spectral information puts forward higher requirements for remote sensing image classification. Many hyperspectral image classification algorithms based on traditional methods, such as superpixel methods, extended morphological feature methods, space-spectral joint classification algorithms based on combined kernels, and classification algorithms based on support vector machines and graph cuts, have achieved certain results. In recent years, benefiting from the improvement of hardware conditions and the update and iteration of algorithms, various deep learning methods have emerged one after another, and have been introduced into the field of hyperspectral image classification by researchers, further improving the accuracy of hyperspectral classification. Autoencoder, Convolutional Neural Network(CNN), and capsule neural network have all been experimentally verified to be effective in this field. Different from the common two-dimensional convolutional neural network, the convolution kernel of the three-dimensional convolutional neural network (3D CNN) is a cube, which can naturally integrate the features of spatial dimension and spectral dimension, and has achieved state-of-the-art performance in the field of remote sensing image classification. Conventional 3D CNN usually extracts data cube features from a single scale, which often loses certain local information; excessively increasing the depth of the model will lead to overfitting problems; limited by actual conditions, it is often difficult to obtain hyperspectral data with a large number of labels, while conventional 3D CNN does not perform well for few-sample situations (for example, the total sample size is only a few hundred). To solve these problems, a multi-branch 3D CNN is proposed in this paper, and the three branches are designed with three different 3D CNN structures. For the input hyperspectral data image set, this paper first utilizes the principal component analysis method to reduce the dimensionality of the data, and the dimensionality of the spectral dimension is selected as 40 after dimensionality reduction. The data cube is decomposed into many 19×19×40 image patches, and the label of its central pixel is used as the label of the image patch, and then the method of rotating 90 degrees, 180 degrees, and 270 degrees is introduced for data augmentation. In the feature extraction stage, a three-dimensional convolutional neural network connected in parallel with three branches is employed to extract features from three spatial scales of 2×2, 4×4, and 6×6. In the training phase, Adam optimizer is used to optimize the parameters of the three branches, respectively, and the cross-entropy loss function is adopted. In order to alleviate overfitting, the dropout unit and Batch Normalization are introduced. In the test phase, the features extracted from the three branches are combined by weighted connection, and the optimal weighting coefficient is optimized by utilizing simulated annealing algorithm. In terms of classifiers, the logistic regression classifier is adopted, which has performance not inferior to fully connected neural networks for small-sized and medium-sized data sets. In order to verify the effectiveness of the method in this paper, 10% of the labeled data were used for training on public datasets such as Indian Pines, Pavia University, and Salinas, the overall accuracy of 98.60%, 99.83%, and 99.97% were respectively obtained. Our method outperforms the comparative methods such as support vector machine, 2D CNN, and conventional 3D CNN. Moreover, the overall accuracy of the method in this paper is studied when the amount of data in the training phase gradually decreases, and compared with the single-branch 3D CNN with data augmentation, multi-branch 3D CNN without data augmentation, single-branch 3D CNN without data augmentation. The method in this paper is also compared with the DAMA and DBDA methods in the case of a few samples. Compared with the comparison method, the performance of the method in this paper still maintains a high classification accuracy when the sample size is small. In addition, a practical test was carried out using the method in this paper. A series of experimental results show that compared with various comparison methods, the method proposed in this paper has a good classification accuracy and has high application value for hyperspectral image classification problems.