Accurate infrared small and weak target detection is the key to real-time monitoring, tracking, and guidance. Infrared weak and small targets have problems of high detection difficulty, high false detection, and serious missed detection. In this paper, an ultra-lightweight infrared dim small target detection algorithm SL-YOLO is proposed to improve the real-time performance and detection accuracy of infrared dim small target detection algorithms. Firstly, the downsampling scheme is redesigned to adjust the network architecture for the infrared image feature information to solve the problem of feature gradient reduction and feature disappearance for infrared weak targets. Then, a network model pruning algorithm is designed to integrate pruning algorithm with network structure, removing redundant parameters, and improving detection speeds. Finally, the SIoU Varifocal loss function is designed to equalise the positive and negative samples with overlapping losses while weighting the positive samples to solve the problem of background interference. The experimental results show that the detection accuracy is improved to 96.4% and 98.1% under the SIRST and IDSAT datasets, respectively. The model volume and computational complexity can be compressed to 190 kB and 0.9 GFLOPs, and the inference speed is reduced to less than 3 ms. Comparing with the mainstream algorithms, the improved algorithm has achieved good results in terms of detection accuracy, model volume; computational complexity. It can meet the real-time detection requirements.