In order to realize the clinical detection of hepatocellular carcinoma (HCC) in vivo, real time and earlier, a normal liver cell line L02, a low-metastatic-potential hepatocellular carcinoma cell line MHCC97-L and a high-metastatic-potential hepatocellular carcinoma cell line HCCLM3 were measured, respectively, based on the established fiber confocal back scattering micro-spectrometer (FCBS). The principal component analysis (PCA) and the support vector machine (SVM) algorithm were used to classify the acquired spectrums, respectively. The PCA was used to study the spectrum in wavelength range of 500-900 nm. The first two of the principal components have taken 95.4% of the whole information; therefore, the three kinds of cell distribution were distinguished obviously on the scores diagram of principal component. 69 object data were chosen randomly to train the SVM classification model. 50 sets of these data were used as training sets and 19 sets were used as testing sets. The classification accuracy of the model has reached 94.7%. These results have indicated that the back-scattering micro-spectra of cells measured by fiber confocal back scattering micro-spectrometer (FCBS) combined PCA or SVM could classify liver cancer cells with different metastatic potential automatically. This will provide the necessary testing tools for the research of hepatocellular carcinoma cell in vivo and real time.