The tracking method based on a people model contributes to realizing the kinetic people tracking for a given video,and it can learn the people model by using much less frames of the video. This paper proposes a learn algorithm for learning the people model in the given video.By using a tree pictorial structure model to represent the detuned generic people in the video, and a modified Hidden Markov Model(HMM) to simulate the motion of people between the two frames of the video, a machine learning method is used to the modified HMM to obtain the estimation of parameter of the modified HMM,and to capture the people model from the video. The learned model consists of different body templates covered with color information. For learning the color of the local parts of the people model by proposed algorithm, an instance-specific model has been obtained. The experiment demonstrates that the kinetic people by proposed algorithm model can be learned with sequences images of 80-90 frames involving people motion,which shows the learning method works well for learning the kinetic people model based on video, and can rapidly learn people models for one or more persons in the video.