In order to improve the robustness of correlation filtering (CF) tracking algorithm, and overcome the problems that the traditional CF method cannot handle target scale change and does not use image color feature, a scale adaptive tracking algorithm is proposed based on correlation filtering improvement with fused color features. Firstly, the target searching area of the image is transferred from the color space of the three primary colors (RGB) to the Lab color space to obtain the Lab three channel features of the search area. Then, Lab color features and histogram of oriented gradients (HOG) feature are fused to obtain the image feature of multi-channel. The kernelized correlation filtering (KCF) is used to get the output response chart and find the position of maximum response, namely target location. Finally, the scale model is established through the Lab color feature, and the different scale image blocks are intercepted from the current frame target position. Optimal estimation of the target scale is obtained when we compare the scale image blocks with scale models. 35 pieces of open color video sequences are selected in experiments for testing, and the proposed method is compared with five other tracking methods with excellent performance. Experimental results show that the proposed method is well adapted to the phenomena of target occlusions, deformation and scale change in color video sequences,and its average performance outperforms the other compared methods. At the same time, the real-time tracking speed of the proposed method is 76 frame·s -1.