To solve the problem of low recognition rate and high false alarm rate in the study of small pedestrian target detection in infrared image, this paper studies YOLOv3, one of the best target detection algorithms, and based on it proposes a small pedestrian detection algorithm that meets real-time requirements. Based on the fact that the classification accuracy is still insufficient in YOLOv3, this article studies the idea of feature reweighting from SENet, and introduces the SE block into YOLOv3, which improves the feature modeling ability of the network. The feasibility of the algorithm is verified by experiments with infrared images collected in actual complex scenes. The experiment results show that the improved network has higher accuracy and lower false alarm rate in small pedestrian detection task, and the algorithm maintains real-time characteristics of the original algorithm.