In order to solve the problem that the cumulative error caused by model updating on mean shift object tracking becomes larger in the following tracking, a mean shift object tracking method based on four channel non-separable wavelets is presented. Based on the decomposition of target image by non-separable wavelets, the accurate target region is segmented by using high frequency sub-images. The high and low frequency characteristic values of this region are fused, and mean shift tracking is carried out. During tracking, the scale and model updating based on the target contour are used, and the adaptive updating of target feature model is carried out by using correlation coefficient of the sub-features. Results show that the proposed method has both real-time ability and accuracy in tracking the change of scene and target shape. Compared with the tracking methods without image segmentation, the proposed method has better tracking accuracy. Compared with the tracking method using conditional random field (CRF), it has better processing speed and accuracy.