In this study, the You Only Look Once Version 3 (YOLOv3) target recognition algorithm in deep learning technology is used to identify the ship in an infrared image collected using an infrared imager from the sea surface. The infrared imager captures images at a frequency of up to 50 frames per second. To reduce network computing time, a few ideas are generated based on YOLOv3; additionally, a full convolution structure and the LeakReLU activation function are used to redesign a lightweight basic network to accelerate detection. The output layer uses the softmax algorithm to regress according to the characteristics of the collected infrared images, which improves the detection speed and accounts for detection accuracy.