Granulation is a common internal disease of citrus fruits, and it is difficult to identify the fruits with this disease from their external features. In this study, an acoustic vibration experimental setup was constructed using a micro-laser Doppler vibrometer (micro-LDV) and a resonance speaker. This was used to collect vibration response signals of ‘Aiyuan 38’ jelly orange. The one-dimensional vibration response signals were converted into vibration multi-domain images, and a Resnet-Transformer network (ResT) was constructed to extract deeper features from the vibration multi-domain images for identifying granulation disease in jelly oranges. In this paper, the ResT, Resnet50, and Vision Transformer (ViT) models were trained using vibration multi-domain images, and their performances were compared. Then, partial least squares discriminant analysis (PLS-DA) and support vector machine (SVM) models were trained using vibration multi-domain image texture features or vibration spectrum features, and the performance was compared with the ResT model. The results show that the ResT model trained using vibration multi-domain images can achieve accurate identification of jelly orange granulation disease with detection accuracy of 98.61%, model F1 of 0.986, precision of 0.986, and recall of 0.986. The proposed method can accurately identify granulated jelly oranges with simplicity, fast speed, and low cost.