To solve the low accuracy of target detection caused by low contrast, insufficient features, and unclear details in complex infrared scenes, an improved infrared dim target detection method based on YOLOv5s model was proposed by creating a two-way convolution and Concat (TCC) module and introducing the Huawei Ghost module. Firstly, combining the low-level semantic features of infrared images, a TCC module was created using two convolution and multi-scale thought, which improved the comprehensiveness of feature extraction. Then, to simplify the network structure and reduce the number of parameters, a lightweight Ghost module was introduced to improve the SPP pooling layer and CSP2 convolutional network. Finally, using unmanned aerial vehicles (UAVs) as experimental objects, a dataset of infrared dim targets was constructed under various meteorological conditions during day and night, verifying validity of the improved algorithm. The results show that the detection accuracy of the improved YOLOv5s model is increased by 1.34%, and the mean average precision (mAP) is increased by 2.26%, which is superior to YOLOv4-tiny and YOLOv7-tiny models. It has the same accuracy as YOLOv8s model, but the number of model parameters is only 53% of the YOLOv8s model, which fully meets the needs of embedded device deployment.