In the process of infrared target tracking, the imaging area of infrared targets is usually small due to changes in target distance and perspective, resulting in a lack of sufficient pixel information in the infrared image, and making it difficult to accurately extract target features and increasing the difficulty of target tracking. Therefore, a long-distance multi-scale infrared target tracking technology based on anchor boxes is proposed. First of all, the tracking block diagram and the real block diagram are corresponded by encoding to get the coordinate value of the center of the target frame. To ensure the accuracy of the calculation, a feature fusion threshold is set to determine the high overlap between the extracted information and the true information, and the pixel grid is divided according to the edge, center, and vertex coordinates of the anchor box, and a classification output vector is output based on the feature values to complete the infrared target feature extraction. As a result, a loss function is used to provide the category loss of anchor boxes, target boxes, and actual boxes, as well as different category loss functions such as candidate boxes. According to the voxel values of different points in the grid, the candidate box and real box data are compared one by one in the form of coding, and the long-distance multi-scale infrared target tracking is achieved through iteration. The experimental results show that the proposed method has good recognition performance for long-distance multi-scale infrared targets, and the recall curve is basically maintained above 0.9. It indicates that the proposed method has good long-range multi-scale infrared target tracking performance.