Aiming at the problems of insufficient feature extraction, incompleteness and low recognition accuracy in action recognition based on human skeleton sequence, a action recognition model based on multi-branch feature and multi-scale spatio-temporal feature is proposed in this paper. Firstly, the original data are enhanced by the combination of various algorithms. Secondly, the multi-branch feature input form is improved to multi-branch fusion feature information, which is input into the network, respectively. After a certain depth of network modules, it is fused together. Then, a multi-scale spatio-temporal convolution module is constructed as the basic module of the network to extract multi-scale spatio-temporal features. Finally, the overall network model is constructed to output action categories. The experimental results show that the recognition accuracy on Cross-subject and Cross-view of NTU RGB-D 60 data set is 89.6% and 95.1%, and the recognition accuracy on Cross-subject and Cross-setup of NTU RGB-D 120 data set is 84.1% and 86.0%, respectively. Compared with other algorithms,the more diversified and multi-scale action features are extracted, and the recognition accuracy of action categories is improved to a certain extent.