A new algorithm to estimate the scale and orientation of a tracking window is presented. The algorithm overcomes the problem that the traditional Mean Shift based tracking algorithm often fails when a deformable target is tracked because of the rigid tracking window. Firstly, the principal components of the variance matrix are adopted to compute the scale and orientation of tracking target, then the similarity measure and Kalman filter are used to update the tracking window. Experimental results show that the algorithm can be implemented in real-time and can adapt to changes of scale and orientation of the moving object. The average computing time is only 17.45 ms/frame with the object's scales varying between 35 pixel×17 pixel and 176 pixel×80 pixel. This algorithm can satisfy the requirements for tracking non-rigid objects.