There are huge challenges in how to comprehensively utilize multi-modal data from ship monitoring in complex sea area scenarios for efficient feature extraction and feature fusion to comprehensively improve ship identification accuracy. Aiming at the problem of ship identification accuracy from a single data source in the maritime environment, an effective ship identification algorithm for multi-modal data feature extraction and feature fusion is proposed, and then feature fusion is performed based on a deep residual network model to improve the accuracy of ship identification rate. Through comparison of experimental results, compared with other algorithms, the average accuracy of the ship identification algorithm based on multi-modal data is increased by about 18%, which effectively improves the accuracy of ship identification and has reference significance for research and development in related ship fields.