Coffee is one of the top three beverages in the world, and it is crucial to strictly control the quality of coffee. Using three different varieties of coffee beans as the object, the terahertz time-domain spectroscopy combined with chemometrics is used herein to identify three different coffee beans quickly. Various pretreatment methods are used to reduce the experimental errors, and principal component analysis is used to reduce the dimensions of the spectral matrix. The dichotomous model based on partial least squares discriminant analysis (PLS-DA) is established, and another model based on a support vector machine (SVM), a back propagation neural network, and a random forest multiclassification discriminant is then established. The PLS-DA dichotomous model exhibits the ideal qualitative discrimination effect whose accuracy is 98%. Among various classification models, the SVM model based on baseline correction is the most effective model, with the total accuracy reaching 98%. This study shows that it is feasible to use terahertz spectroscopy to quickly identify coffee bean varieties and a better support vector machine model is established based on baseline correction, which provides an empirical reference for the qualitative detection of other agricultural products using the terahertz time-domain spectroscopy.