Aiming at the problem of sea-sky-line detection in low-contrast infrared images being difficult and easily affected by interference factors such as clouds, strip waves and sea clutter, we propose a method of using polarization difference images for sea-sky-line detection. Firstly, Polarization Difference Imaging (PDI) is used to enhance the local contrast of the sea surface area and the Signal-to-Noise Ratio (SNR) of the sea-sky-line. A large-scale local contrast accumulation method of the polarization difference images is then used to determine the sea-sky-line area. Finally, the accurate detection of a small-scale sea-sky-line is completed by combining the gradient significance and polynomial fitting in the sea-sky-line area. Overall, the methodology integrates multi-dimensional information such as the Degree of Linear Polarization (DOLP) and the Angle of Polarization (AOP) for sea-sky-line detection, and combines large-scale and small-scale detection, which can effectively overcome interference of factors such as clouds, strip waves and sea clutter. The experimental results show that the accuracy of this algorithm for sea-sky-line detection is 98.5%, and the average time consumed is 16 ms. The experimental results indicate that the proposed algorithm can realize fast and accurate sea-sky-line detection so it has wide applicability in different scenes.