In order to improve the robustness of the target tracking algorithm in complex scenes and optimize the operating efficiency of the model, a correlation filter tracking algorithm based on adaptive feature fusion was proposed. The algorithm adopts histogram of oriented gradient feature and deep feature extraction network feature to construct the target information, uses the peak to sidelobe ratio and the value of side lobe ratio of feature response to adaptively determine the fusion coefficient, and predicts the target position according to the fusion response. In order to reduce the influence of illumination variation, occlusion and target deformation on the tracking process and adapt to the change of scene, the average peak-to correlation energy was introduced to design the filter learning rate adjustment mechanism and update the model dynamically. Through the lightweight design of the deep feature extraction network, the parameters of the feature network were reduced and the tracking speed was improved. Experimental results show that the algorithm effectively reduces the influence of interference on the tracking results, and the algorithm has better performance in tracking precision, success rate and speed compared with other tracking algorithm on the public video dataset OTB100.