Aiming at the problem of tracking failure and less robustness caused by background clutter, occlusion and object deformation in infrared object tracking, an infrared object tracking method combining tracking-learning-detection method and correlation filtering theory was proposed. Based on the traditional correlation filtering framework, the proposed method combines the direction gradient histogram feature and the luminance histogram feature to improve the model drift caused by slight deformation of the target. Aiming at the multi-peak response problem caused by background clutter and occlusion, the response of the target background area was punished, and the multi-modal detection mechanism of target and background response was established to achieve the target from coarse to fine positioning, and the adaptive learning rate was used to optimize the drift problem of the tracking model; Aiming at the problem that the object was severely occluded or the object was out of view, the global re-detection of the target was implemented to achieve the target re-capture. The experimental results show that the proposed algorithm effectively solves the object loss caused by background clutter and occlusion in the complex infrared ground environment. Based on the benchmark OTB-2015 and infrared video sequence test, compared with the mainstream correlation filtering tracking algorithms, the proposed algorithm improves the tracking accuracy and success rate by 5.6% and 4.1% respectively compared with the Long-term Correlation Tracking (LCT) algorithm; In the occlusion index test, the proposed algorithm improves the tracking accuracy and success rate by 4.6% and 6.1% respectively compared with the LCT algorithm.