A lightweight infrared image target detection algorithm GPNet is proposed to address the need for accurate and real-time target detection in resource-constrained infrared imaging systems. The feature extraction network is optimized using GhostNet, feature fusion is performed using an improved PANet, and a depth-separable convolution is used to replace the ordinary 3×3 convolution at specific locations to better extract multi-scale features and reduce the number of parameters. Experiments on public datasets show that the algorithm in this paper reduces the number of parameters by 81% and 42% compared with YOLOv4 and YOLOv5-m, respectively; the average mean accuracy is improved by 2.5% and the number of parameters is reduced by 51% compared with YOLOX-m; the number of parameters is 12.3 M and the detection time is 14 ms, which achieves a balance between detection accuracy and number of parameters.