In order to improve the detection probability of a single-frame infrared image and stably detect weak and small targets in image sequences, a weak and small infrared target detection algorithm under complex sky background is proposed based on the improved bilateral filtering and polynomial fitting. The background correlation factor is introduced into the weight coefficient of the traditional bilateral filtering algorithm to effectively reduce the influence of the target point during background suppression, improve the signal-to-noise ratio of the target area and the detection rate of the single frame image. In order to further eliminate the false target, the motion features of the fusion targets are combined to perform multi-frame confirmation on the target point. Aiming at the target miss detection caused by target flicker in sequence detection, the polynomial fitting algorithm is introduced to predict the target position of the next frame, which effectively avoids the problem of truncation of the target trajectory. The experimental results show that the algorithm can stably detect weak and small target trajectories under complex sky backgrounds when the signal-to-noise ratio is less than 2.