To solve the problem of target tracking in the presence of background noise, occlusion, deformation and scale variation, a context-aware tracking algorithm based on a visual saliency and perturbation model was proposed. First, the proposed algorithm was based on the correlation filtering algorithm. The contextual information of the target was introduced into the classifier learning process. The context-aware correlation filter was then constructed, which improves the robustness of the algorithm. Meanwhile, the histogram perturbation model was introduced. The target response map was calculated using the weighted fusion method to estimate the target position change. Finally, the target saliency map was constructed using visual saliency to solve the target relocation problem under occlusion problem. The scale estimation strategy was used to solve the problem of target scale variation. The algorithm performance was tested using open-source datasets and was compared with eight popular tracking algorithms. The experimental results demonstrate that the accuracy and success rate of the algorithm are 0.695 and 0.708, respectively, which are better than other algorithms. Compared with the traditional correlation filtering algorithm, the proposed algorithm can solve the target tracking problem with complex background noise, occlusion, deformation and scale changes. It has a certain theoretical research value and practical value of engineering.