The development of observation technology has led to massive spectral data. How to automatically classify these data has received attention from researchers, the most important of which is feature extraction. Given the limitations of manual processing, most of the research uses machine learning algorithms to extract feature-based spectral data. However, these machine learning algorithms cannot handle massive spectral data due to the high spatial and temporal complexities. The pre-trained models emerging in recent years have excellent feature extraction capabilities. Still, there is little research on the effectiveness of such a model in the feature extraction of spectral data. Therefore, this paper takes the stellar spectral data as the research object separately introduces the pre-training models such as BERT, ALBERT, GTP, and Convolutional Neural Networks (CNN) for feature extraction and classification of the stellar spectral data, and tries to verify the effectiveness of these pre-training models for feature extraction of stellar spectral data by comparing the experimental results. Python programming language is used to write the spectral classification program. Based on the feature extraction of the pre-trained models, the CNN model in TensorFlow 1.14 is utilized for spectral data classification. The dataset used for the experiment is the SDSS DR10 stellar spectral dataset, including K-type, F-type, and G-type. The grid search and 5-fold cross-validation are utilized to obtain the experimental optimal parameters. The BERT model has the highest classification accuracies compared to ALBERT and GPT with the same experimental conditions. In terms of the average classification accuracies, the average classification accuracies of the BERT model are 0.025 1, 0.021 5, and 0.022 5 higher than that of ALBERT, and 0.049 7, 0.042 4, and 0.043 2 higher than that of GPT, on the K-type, F-type, and G-type stellar datasets. It is easy to draw the following conclusions by analyzing the experimental results: Firstly, the classification accuracies improve with the scale increase of training data; Secondly, the same model has the highest classification accuracies on the same training dataset of K-type stellar, followed by the F-type and the G-type; Thirdly, the BERT model has the best ability of feature extraction compared with ALBERT and GPT.