Aiming at the problems of low behavior discrimination ability and misjudgment caused by different change speeds of actors and action states and the lack of correlation research between actions in action recognition tasks, a temporal correlation attention mechanism model based on SlowFast architecture was proposed. Firstly, the optical flow was abandoned and the video data was directly used as the network input, so that the model could be trained end-to-end. Secondly, a temporal correlation attention mechanism composed of correlation attention and temporal attention was defined. The correlation attention mechanism was used to extract the correlation information between actions, and then the information was input into the temporal attention mechanism to suppress useless features. Finally, to solve the problem of the loss of correlation between features caused by the large step size of the convolution kernel in the path fusion process of SlowFast, a more effective continuous convolution operation was proposed. Experimental results on UCF101 and HMDB51 datasets show that the proposed method has advantages in accuracy and robustness compared with the existing methods.