Taking the infrared and visible image of marine targets as data source，considering the characteristic of multiscale and multi-band information of data source for marine targets，based on the prototype YOLOv3 network architecture and FPN principle，the bottom feature map of the 11th layer is fused with the deep layer feature map of the 103rd layer to achieve the expansion of the network scale，and k-means clustering algorithm is used to obtain the prior box at a more refined scale. Meanwhile，the infrared and visible images are combined in a certain proportion to form the physical layer fusion of the image source，and then the mixed data set is constructed for multi-band cooperative model training. The experimental results show that the recognition accuracy of S4-YOLO network model is higher than that of YOLOv3 and YOLOv3-Tiny models，and it can adapt to the identification needs of multi-scale marine targets.