To solve these problems, such as large amount of calculation, lack of ability of generalization and poor detection performance, a light-weight night driving infrared image target detection algorithm is proposed in this paper. The algorithm first utilizes the Ghost structure as the backbone network to reduce the amount of model calculation. Then, the bidirectional feature gramid network (BIFPN) structure and coordinate attention (CA) mechanism are introduced in the neck to improve the model detection effect. Finally, the Focal-EIOU and Mish functions are used as the loss function and activation function of the algorithm to improve the convergence speed and regression accuracy. The experimental results show that the improved algorithm has significantly improved compared with YOLOv3-tiny in all aspects. Compared with YOLOv5, the accuracy has increased to 88.9%, the model volume has been reduced by 24.09%, the number of parameters has been reduced by 25.07%, and the amount of calculation has been reduced by 28.48%, the detection accuracy is improved in the two categories of person and bicycle. A balance between detection accuracy and model complexity is achieved.