In this study, we propose a robust adaptive tracking algorithm for infrared dim objects that addresses the problem of tracking discontinuities and failures caused by missed detections and clutter in complex scenes. In the pre-processing stage, an algorithm that measures the image complexity eliminates unnecessary calculations. This algorithm determines the scene type by calculating multiple features of the infrared image to obtain the scene complexity, and then selects the corresponding detection algorithm to extract the target candidate location, grayscale and local histogram features. Subsequently, a measurement model and likelihood function are established based on the scene type. In the tracking stage, to flexibly match the filtering parameters of the generalized labeled multi-Bernoulli (GLMB) filter, an adaptive algorithm suitable for video image distribution is proposed for track initiation. Aiming at the unknown detection probability of an infrared image sequence, a cardinality probability hypothesis density (CPHD) filter was integrated into the GLMB to estimate the detection probability of the target in real time, thereby improving the accuracy of the tracker. The simulation results show that the proposed algorithm can effectively track small infrared objects in different complex scenarios.