Human action recognition in the video sequence have become a hot research topic in computer vision field. In order to extract the contour feature of the human’s behavior sequence more effectively, a new algorithm for human action recognition based on Local Binary Pattern (LBP) is proposed. Firstly, background subtraction algorithm is used to extract the complete human motion sequence in the video, and the LBP operators are used to calculate the samples’ LBP feature space which is composed of the motion sequences’ LBP feature spectrum. Then, the behavior feature is generated by k-means clustering method. Finally, the Hidden Markov Model (HMM) is adopted for the classification. During the experiment, the test experiment is performed in the two public behavior databases respectively, and the average recognition rate can reach more than 85%. The intersection of the two databases experimental results shows that the proposed algorithm has certain robustness.