The Forward-looking Infrared (FLIR) camera is installed in the front of ship. In the FLIR surveillance video sequence used for moving ship detection, it can be achieved to avoid ship-ship collision so that the safety sailing is ensured. Ship detection and tracking algorithm in FLIR images from complicated background along the inland waterway is proposed. Firstly, a method is proposed to extract the sky-water line, and the Region of Interest (ROI) is extracted based on the sky-water line. Then, multi-scale fractal feature related with K (MFFK) is presented to detect ship. Finally, an improved mean shift algorithm for moving ship is proposed to track ship in inland waterway. Experimental results have shown that the approach is robust and achieves real-time and reliable ship detection and tracking.