Loquat is a freshwater fruit at the turn of spring and summer, it has a sour taste and can be eaten directly or made into candied fruit or wine, and it has the effect of resolving phlegm, relieving cough, harmonizing the stomach and lowering gas. The texture of loquat is soft and juicy, so it is prone to be bruised during picking, storage and transportation, resulting in economic losses. Therefore, detecting bruised loquats with high precision and rapid classification is essential. Meanwhile, we have used different methods to treat loquats with different bruising levels to reduce economic losses. The ones with light bruises can make loquat juice and paste. The ones with moderate bruises can be removed from damage region to make canned loquats for preservation. The ones with heavy bruises can be disposed of directly to save storage costs. At present, the bruise level of loquats is mainly discriminated by the operator’s naked eye. It is affected by personal habits, light intensity and subjective psychological factors, which will cause misclassification. In this paper, we propose a method based on hyperspectral imaging technology spectral combined with color features to classify loquat bruise level with high precision, rapidity and non-destructiveness. Firstly, we used a free-fall collision device to prepare light, moderate and heavy bruised loquat samples and used a hyperspectral imaging system to collect data. Secondly, we select the average spectrum of 100 pixels in the region of interest as the sample spectrum and preprocess the spectrum with MSC, which is used as the spectral feature for the subsequent model. Finally, we combined spectral features with color features and used RF, PLS-DA, ELM, and LS-SVM to build loquat bruising level models based on spectral features, RGB color features combined with spectral features, HSI color features combined with spectral features, and mixed color features combined with spectral features, respectively. Among all the above models, the loquat bruise level model based on mixed color features combined with spectral features has the best prediction effect. The overall recognition accuracy of the models using RF, PLS-DA, ELM and LS-SVM algorithms is 91.11%, 86.67%, 95.56%, and 100%, respectively. The RBF-LS-SVM bruising loquat model has the highest accuracy. The results show that the model based on single spectral features has the lowest accuracy, the model combined with RGB or HSI color features has higher accuracy, and the model based on spectral features combined with mixed color features has the highest accuracy. This study provides a certain theoretical reference and experimental basis for fruit bruising level discrimination.