In this paper, an improved infrared small target detection model, infrared-YOLOv5s, based on YOLOv5s is proposed to address the problems of low resolution, complex background and lack of detailed features of infrared images. In feature extraction stage, SPD-Conv is used for down-sampling, which divides the feature map into feature sub-maps and concatenate them by channel to avoid the loss of features caused by down-sampling in the process of multi-scale feature extraction. And an improved atrous spatial pyramid pooling module is designed to improve feature extraction capabilities by fusing features with different receptive fields. Then, in feature fusion stage, a deep-to-shallow attention module is introduced to embed deep semantic features into shallow spatial features to enhance the expression of shallow features. Moreover, in prediction stage, the prediction layers, feature extraction layers and feature fusion layers for large target detection in the network are cut down to reduce the model size and improve real-time performance at the same time. The effectiveness of each module is verified by ablation experiments, and experimental results show that the proposed model achieves 95.4% mAP_0.5 of on SIRST dataset, which is 2.3% higher than that of original YOLOv5s. The model size is reduced by 72.9% to 4.5 MB, and the inference speed on Nvidia Xavier reaches 28 f/s, which is conducive to the actual deployment and application. Therefore, the effectiveness of the proposed model is further verified by transfer experiments using Infrared-PV dataset, and the proposed model can meet the real-time requirements while improving the performance of small target detection in infrared images, and is suitable for the task of real-time small target detection in infrared images.