For better accuracy of mixture component identification and less complexity of modeling, a progressive detection system based on surface-enhanced Raman spectroscopy with rough classification followed by subdivision is constructed in this paper. First, a characteristic peak discrimination algorithm is used to extract features from all samples and establish a rough classification model, according to which substances are classified into single-component and multi-component ones. Then, automatic extraction of spectral characteristics is accomplished through normalization and principal component analysis. A subdivision model is developed upon a multi-output least squares support vector machine. Finally, the particle swarm optimization algorithm is employed to optimize the parameters so as to achieve an accurate prediction of the composition of multi-component samples. Experiments are carried out with Rhodamine 6G, Nile blue and crystal violet as probe molecules. The results show that the characteristic peak discrimination algorithm extracts sample features with an accuracy of 99.44%. The rough classification model correctly identifies all 90 samples in the blind test. Moreover, the correlation coefficient of the subdivision model for multi-component sample identification is not less than 0.995 and the root mean square error is not more than 2.67343%. Enabling both qualitative and quantitative detection of samples, the Raman detection system proposed in this paper provides an effective identification method for future detection of complex substances such as drugs.