Considering the problem in which the existing thermal infrared target tracking algorithms have difficulty dealing with similar object interference and target occlusion, the multi-task framework in the MMNet algorithm is introduced to obtain the specific discriminant features and fine-grained features of thermal infrared targets, which are fused to identify thermal infrared objects between and within classes. In addition, the peak side-lobe ratio is adopted to dynamically set the model update parameters and obtain the target change information more efficiently, in addition to evaluating the tracking results. For unreliable tracking results, a Kalman filter was unutilized to predict the target. The experimental results on the LSOTB-TIR dataset demonstrated that the performance of the improved algorithm was optimal. Compared with MMNet, the tracking accuracy and success rate were improved by 5.7% and 4.2%, respectively. It can effectively address the challenges of occlusion and deformation and can also be applied to the field of infrared target tracking.