With the rapid development of hybrid rice breeding technology, hybrid rice varieties are becoming increasingly diverse, and their quality and price vary widely. The use of intelligent means for rapid classification, grading and quality detection of hybrid rice seeds has become a hot spot in hybrid rice research. In this paper, we first investigate the effect of different preprocessing methods on the accuracy of a 1D Convolutional Neural Network (1D-CNN) classification model constructed based on the near-infrared spectra of 10 hybrid rice seeds. The results show that the overall validation and prediction accuracy can be up to 95.4% and 92.9% respectively when the near-infrared spectra are preprocessed with the Savitzky-Golay convolution smoothing algorithm (SG). Secondly, the three most important feature wavelengths were selected by the random forest feature wavelength selection algorithm to build a single-wavelength grayscale image dataset and a 3-wavelength reconstructed pseudo-color image dataset, and the hybrid rice seed classification model based on the convolutional neural network VGG and the residual network ResNet of the image dataset was constructed and studied. The results show that the VGG model based on the pseudo-color image dataset can obtain the optimal classification effect, and the classification accuracies of its validation set and test set are 92.8% and 92.8%, respectively. Compared with the ResNet classification model based on the pseudo-color image dataset, an improved value of 3.6% is achievedin the validation set and 4.9% in the test set. In order to further improve the classification accuracy, a hybrid rice seed classification method based on the fusion of image information and spectral information is proposed. This methodextracts spectral features using the 1D-CNN network branch and extracts dimensionalspatial features using the 2D-CNN network branch. 2Branches-CNN model is then constructed based on the fusion of image and spectral features, and the classification accuracy reaches high values of 98% and 96.7% for the validation set and test set. The classification effect of the 2Branch-CNN model for each type of hybrid rice seeds is also evaluated by calculating the confusion matrix. The results of this paper show that the classification accuracy of the convolutional neural network model can be effectively improved by image-spectrum fusion, and the construction of a two-branch convolutional neural network model based on image-spectrum fusion will provide new ideas for rapid screening and classification of hybrid seed varieties.