Pedestrian recognition based on infrared images is an important component of modern security systems. In scenarios with limited computing resources, it is often difficult to balance the detection accuracy and deployment difficulty due to the influence of model size in infrared pedestrian detection algorithms. In response to this issue, a lightweight object detection algorithm based on YOLOv5s is proposed in this paper. Firstly, the MobileNetv3 lightweight feature extraction network is introduced and deep separable convolution is used to reduce the model size, making it easier to deploy to CPU devices. Secondly, the nearest neighbor interpolation upsampling method is replaced with CARAFE (Content-Aware ReAssembly of FEatures) which significantly improves the image reconstruction effect. Finally, EIOU Loss is used as the loss function of the bounding box to improve the regression performance of the model. Additionally, tests are conducted on the sampled LLVIP infrared pedestrian image dataset and the results show that for pedestrian targets in infrared images, the model size is reduced by 80.6% and the number of parameters is reduced by 82.8% while maintaining a high detection accuracy (AP=95.4%); and the inference speed is improved by 43.3% when using a CPU platform for inference, and the performance of detecting multi-scale targets is improved. The above two results validate the effectiveness of the algorithm.