In order to improve classification accuracy of hyperspectral image based on the joint sparse representation, we propose a classification algorithm based on neighborhood similarity. Compared with conventional joint sparse representation algorithm, the weight of different feature categories pixels to pixel P to be test in neighborhood is different. Similarity threshold can be set based on the similarity of all pixels in neighborhood and pixel P. Category of pixel P can be obtained by joint sparse representation pixels which have high similarity with pixel P. And then the spatial information is used to modify classification algorithm, which associates with the categories of the neighboring pixels and gets smooth classification results. Experiment results demonstrate that the proposed algorithm has higher classification accuracy and more stable results.