In order to solve the problems of limited parameter redundancy deployment and limited detection speed in the study of infrared anti-interference in deep learning, this paper proposes a lightweight method based on improved YOLOv5s, which uses Dy-MobileNetV3 to replace the backbone part of YOLOv5s. Dy-MobileNetV3 replaces the SE channel attention mechanism in the lightweight network MobileNetV3 with the CA coordinate attention mechanism to improve the accuracy and computing efficiency, and uses the dynamic convolution module to reconstruct the MobileNetV3 network to maintain the lightweight computing cost and improve the accuracy. Adaptive label smoothing is introduced to reduce the overfitting of the model and improve the generalization ability of the model. Compared with the basic model YOLOv5s, the proposed DyM-YOLOv5s model reduces the number of parameters by 85%, the image processing speed is 3.5 times that of the original, and the accuracy is increased by 4%.