Ship detection from optical remote sensing images is easily disturbed by complex backgrounds, such as clouds, islands, and sea clutter. In this paper, a novel ship detection method was proposed to solve these problems. First, to solve the change in target size, multi-scale saliency maps were generated using a spectral residual visual saliency model, and the optimal saliency map was adaptively selected using the Gini index. Further, considering the problem of missing detection caused by the threshold segmentation, a two-stage segmentation method was proposed to separate the target and background pixels. The local maximums of the saliency map were then obtained using image expansion, and the k-means algorithm was adopted to determine whether each local maximum belongs to the target pixel or background pixel. The accurate candidate locations were obtained using the local threshold segmentation. Finally, the rotation invariant feature based on the radial gradient transform was introduced to further eliminate false alarm. The experimental results show that the proposed detection method can successfully detect ship targets of different sizes and directions and effectively overcome complex background interference. Additionally, the detection accuracy is 93%, and the false alarm rate is 4%, which are better than other saliency- based ship detection methods.