Aimed at the characteristic of the port target revealed in remote sensing image, a port target detection algorithm based on the invariant linear-moment and the closure of the port region is proposed. Firstly, edge-search method and thresholding are used to realize the segmentation and obtain the binary image of sea and land. Then, a closure is computed, and through the closure of the coastline the port region, its inner coastline is confirmed. The invariant linear-moment of the port’s inner coastline is computed, a feature spectrum consists of the invariant linear-moment and closure is got. The unknown port is recognized based on comparison against feature spectrum to the templets which have been stored anteriorly. Experimental results show that the proposed algorithm can correctly recognize the port target.