Aiming at the problem that the traditional particle filter is divergent and failed in tracking for multiple objects, a multi-object tracking algorithm of the particle filter based on the median shift is presented. The intermediate result after sampling can be analyzed by the method based on the median shift and the subset particles of the object can be obtained, therefore, each object can be estimated optimally, the filter accuracy is improved, so the problem of object entering or leaving and the occlusion can be also solved effectively. The simulation results demonstrate that the algorithm is robust and effective for multiple objects under complex backgrounds.