With the application of deep learning in computer vision, its large amount of data, complex network layer structure, insufficient resources in hardware deployment and high delay have become key problems. This paper, by analyzing the advantages and disadvantages of five representative lightweight networks, proposes a lightweight network improvement based on MobileNet, which applies lightweight networks to infrared target detection field. FPGA is used as the hardware carrier. In this network, Tanh activation function is used to replace the original activation function and the number of network layers is simplified to adapt to the feature extraction of infrared targets. In view of the problems existing in the hardware implementation of deep learning target detection algorithm, such as large amount of data, large resource occupation and high calculation delay, FPGA is adopted for hardware implementation. The experiment shows that on Xilinx Zynq-7020 XA development board, the clock frequency is set to 100 MHz and the input image size is 640×512. The improved MobileNet can achieve each image of 5.1 ms with the same accuracy as the original one.