The difficulty in video tracking is how to find the matching points of target from flame to frame accurately and reliably. Because of the low contrast between infrared target and background, the blurred edge and low dynamic range of grey lever, the infrared target tracking is more difficult than visible target. An infrared target tracking algorithm is proposed based on histograms and moment invariant. We use the mean shift algorithm based on histogram to calculate the suboptimal matching point rapidly and efficiently. Because the histogram does not contain the target′s shape features, the suboptimal matching point always has some errors. These errors should be amended to avoid accumulating tracking-error and the tracking-point drifting away from the object gradually. So moment invariant is used to modify these errors and improve the tracking stability and accuracy. Experimental results show that this algorithm is able to eliminate drifting phenomenon and enhance the tracking stability and accuracy.