In recent years, convolutional neural networks have made remarkable progress in the field of hyperspectral image classification, but they can only perform regular grid operations on images, and cannot adaptively perform feature aggregation. Therefore, a segmented forest-based multi-scale convolutional neural network hyperspectral image classification method is proposed in this paper, which consists of four steps. Firstly, principal component analysis is used for dimensionality reduction, and a multi-scale segmented forest is constructed according to the spatial information of images to establish the relationship between the subtrees. Then, a U-net model architecture based on graph convolutional network is proposed to establish the transformation of graph structural features between multiple scales by pooling and unpooling. The network uses a graph convolutional neural network to perform adaptive feature aggregation and fuses multi-scale features by layer hopping connection between encoder and decoder. Finally, the semi-supervised classification of nodes is carried out through SoftMax. The experiment is verified on the public hyperspectral dataset, all of which achieves good classification accuracy, demonstrating the effectiveness of the method.