The human tracking is the key in the video surveillance. This text focuses on non-rigid objects, learning based on the HSI color model template. Each pixel of the template is modeled using two components with H and S. Get template from the current frame, statistic the information of every frame, and predict in the next frame using Kalman filter. Decide whether to update the template after comparing the forecast region and the template, reducing the amount of calculation. And for restraining the tracking window, information which is calculated from the color I is introduced. After that, the template and the window will be updated. The experimental results show that the proposed method achieves continuously tracking, and resolve the problem with the object disappeared.