Target model drift is an important factor for visual tracking. To deal with the problem,a novel adaptive tracking algorithm is proposed. The spatial-temporal constraints of the tracking region and non-tracking region in sequence are modeled through a conditional random field (CRF),the parameters of which are updated using online learning method according to the changes of scene. The pixels of all images can be labeled as target or background based on the CRF model. During tracking,the optimal label field and the confidence map are first achieved by resolving the CRF model. Then the similarity measure between the target model and the target candidates,combined with the confidence map,is fed to the mean shift algorithm for the target localization. A selective sampling update strategy is utilized to alleviate the model drift. The experimental results demonstrate the efficiencies of the proposed algorithm in several real sequence testings.