The object tracking method based on Circular Structure with Kernels (CSK) is only suitable for tracking objects with fixed scale.To solve the problem, a new patch-based scale adaptive CSK tracking method was proposed.First, the relationship between the position of patches and the scale of object was derived from spatial structure of each patch.Then, the tracking confidence level of each patch was evaluated, and those with high confidence levels were selected for integration.Therefore, effective estimation was made to the scale of the object through independent tracking of each patch and the integration.The robustness of the method under interference was improved.Experiment was made by using the typical vision sequences.The result showed that the algorithm can not only realize real-time tracking of the change of object scale,but also has higher accuracy and robustness compared with the original CSK method.