As a real-time, non-invasive, and high-resolution imaging method, optical coherence tomography (OCT) provides rich image information using feature extraction algorithms and provides basis for the objective diagnosis of diseases. This study imaged 17 normal thyroid tissues and papillary carcinoma tissues using the OCT system. According to the characteristics of thyroid tissue images, a gray level co-occurrence matrix (GLCM), gray level histogram (GH), center-symmetric autocorrelation (CSAC), and Laws' texture measure (LM) were used to extract the image eigenvalues. Similarly, we quantitatively evaluated the identification performance of the different feature combinations using a support vector machine (SVM) algorithm. The results indicate that GLCM-GH-LM model has the best performance, with a sensitivity, specificity, and accuracy of 96.3%, 92.2%, and 94.3%, respectively. Moreover, it can obtain texture and gray feature information from multiple aspects. This study illustrates that the algorithm based on feature extraction and machine learning can not only provide real-time monitoring images, but also have important reference value for clinical diagnosis of thyroid malignant tumors when performing quantitative analysis and recognition for OCT images of papillary carcinoma of thyroid.