To improve the performance of infrared target detection, weak and small infrared target detection combined with frame difference kernel correlation filtering is proposed. First, the current frame is trained by kernel correlation filtering to obtain the maximum regression value. Then, the difference value is calculated relative to the interval frame to perform a cyclic shift to compensate for the background motion between frames. The relative motion features of the current frame are extracted using the interframe difference method, which enhances the ability to distinguish weak and small targets from the infrared background. Finally, threshold segmentation is performed on the relative motion features to obtain the final detection results. Simulation experiments show that the proposed algorithm effectively detected weak and small infrared targets in complex environments. Compared with similar algorithms, the proposed algorithm suppressed clutter and point-shaped interference sources, and achieved a higher target detection rate. Simultaneously, a large number of operations are placed in the frequency domain, and the operational efficiency is better than that of other algorithms.