Hyperspectral images (HSI) have been widely used in various fields of production and life due to their rich spectral and spatial information. This paper proposes a tensor dictionary learning-based sparse representation classification (Tensor-DLSRC) algorithm, which directly takes the spatial-spectral tensor as the basic unit to exploit the spectral and spatial information and improve the accuracy of hyperspectral image classification. Firstly, the spatial-spectral tensor comprises the spectral vectors of all pixels in the spatial neighborhood of the central pixels. Secondly, the mean vectors of each order fiber of the training spatial-spectral tensor are used as dictionary atoms to generate an initialized dictionary. The tensor-based dictionary learning (TDL) algorithm is proposed to train a set of structured dictionaries from the training samples. Then, a tensor-based sparse representation model is constructed based on the sparsity constraints of the tensor, and the sparse representation coefficient tensor corresponding to the test spatial-spectral tensor is obtained by solving the model. Finally, the class of the test sample is determined according to the minimization of the reconstruction residuals. To analyze the impact of parameters on the classification accuracy of the proposed algorithm, a series of experiments were conducted to discuss the effects of parameters such as training sample size M, neighborhood window size (2δ+1)×(2δ+1), sparsity μ1 in dictionary learning stage, and sparsity μ2 in sparse representation stage on overall accuracy (OA) before conducting classification comparison experiments. To verify the effectiveness of the proposed algorithm, a series of experiments were conducted on three HSIs, (e.g., Indian Pines, Salinas, and Xuzhou) to compare and analyze the classification results of our algorithm with five comparative algorithms: SRC and DLSRC algorithms based on spectral vectors, JSRC and DLSJSC algorithms with added neighborhood spatial information, and Tensor DLSRC algorithm based on spatial-spectral tensor. The classification results were quantitatively analyzed using Average Precision Rate (APR), Average Accuracy (PA), OA, and Kappa coefficients based on the confusion matrix. The proposed Tensor-DLSRC algorithm has the highest average level of OA and Kappa coefficients among the six algorithms. It has the smallest standard deviation, indicating that compared with the comparative algorithms, this algorithm can provide more accurate and stable classification results.