To enhance Camshift algorithm on the adaptability of high-speed object, complex background and covering, an improved algorithm combining inter-frame difference and motion prediction is proposed. First, before calculating back-projection image, the color information is associated with the motion information gained from inter-frame difference to eliminate color noise and automatically initialize tracking. Second, in order to make the algorithm applicable with high-speed object, position of target is predicted and searching window is updated according to tracking state. Experimental results prove that the improved algorithm is robust to high-speed target, covering and color noises.