To improve the accuracy of action recognition based on the human skeleton, an action recognition method based on geometric features and a recurrent temporal attention network was proposed. First, a vectorized form of the rotation matrix was defined to describe the relative geometric relationship between body parts. The vectorized form was fused with joint coordinates and joint distances to represent a skeleton in a video. A temporal attention method was then introduced. By considering the weighted average of the previous frame, a multi-layer perceptron was used to learn the weight of the current frame. Finally, the product of the feature vector and corresponding weight was propagated through three layers of long short-term memory to predict the class label. The experimental results show that the recognition accuracy of the proposed algorithm was superior to that of existing algorithms. Specifically, experiments with the MSR-Action3D and UWA3D Multiview Activity II datasets achieved 96.93 and 80.50% accuracy, respectively.