Since Kernelized Correlation Filters (KCF)tracking algorithm is sensitive to feature selecting and unable to estimate object scale, this paper researches the KCF tracking algorithm based on feature extraction and scale adapting. A scale adaptive KCF tracker by using HHS-OG (Histogram of Hue Saturation and Oriented Gradient, HHS-OG) feature was proposed to improve the tracking performance of the KCF tracker. Firstly, the HSI color space was studied. By taking the complementary of color and gradient in an image, a novel HHS-OG feature focused color and gradient features was proposed to improve the discrimination ability of the KCF algorithm to backgrounds and targets. As the KCF algorithm is unable to process the changed scale, a set of scale factors were used to sample image patches in the detection stage of tracking and the generated corresponding filter response maps were used to estimate the optimal target position and scale. The proposed tracker was tested on a large tracking benchmarks with 50 video sequences. Experimental results show that the tracker runs at a high speed of 37.5 frame per second and has a significantly improvement of 5.4% in representative precision score and 10.1% representative success score. The HHS-OG feature has good discrimination ability for backgrounds and targets and has robustness for target tracking. The scale adaptive strategy is effective for improving tracking performance.