The kernel based tracking has two disadvantages: the tracking window size cannot be adjusted efficiently, and the kernel based color distribution may not have enough ability to discriminate object from clutter background. For boosting up the feature's discriminating ability, both scale invariant features and kernel based color distribution features are used as descriptors of tracked object. The proposed algorithm can keep tracking object of varying scales even when the surrounding background is similar to the object's appearance.