In accordance with complicated object movement and changeable object environment, a multi-pattern fusion algorithm for target tracking is presented. Mean-shift and particle filter algorithms widely applied to target tracking are selected to get tentative locations and Weighted Composite Reference Function (WCRF) is adopted to establish reference model. Then, the distance difference of the tentative locations and the reference model is considered as a criterion to find correct location. Finally, the algorithm updates the reference model according to the distance between reference model and target model in current frame. The experimental simulation results show that the average tracking error of the proposed algorithm is reduced by 50% as compared with that of single target tracking method. If the reference model is updated incorrectly, the probability to find the correct location in the next frame is 67%. After updating the reference model three times, the influence on object tracking is less than 10%,which effectively reduces the tracking error and instability for model updating.