To address challenges in the non-destructive inspection and classification of kiwifruit hardness quality, we propose a classification model that incorporates hyperspectral imaging technology and a convolution neural network. This network combines the spatial feature information extracted by the Haar wavelet and the space-spectrum joint information extracted by the three-dimensional (3D) convolution kernel. In this network, the data decomposition of channel connections is executed to ensure that all features can be utilized by the model, which improves the ability of network feature learning. Experiments on the acquired hyperspectral image-based, self-made kiwifruit hardness quality dataset (named Kiwi_seed) demonstrate that the Haar wavelet transform module can significantly improve the feature extraction ability of the network. Ablation experiments reveal that the classification accuracy of the model incorporating the Haar wavelet transform module is increased by 7.4% and reaches the optimum level at 97.3%, which is better than the classical image classification network. The proposed classification model can be effectively used for the non-destructive inspection and classification of kiwifruit quality.