To improve the performance of person tracking for a mobile robot in complex environments, an adaptive kernel function based Mean Shift algorithm was proposed by using a coarse to fine localization mechanism. In the outer layer, a Radio Frequency Identification Device (RFID) was adopted to detect the person with an ID tag to determine the Region of Interest (ROI) coarsely. In the inner layer, the ROI of a disparity image was processed to estimate an initial searching window. Then, the adaptive kernel based Mean Shift algorithm was applied to location of the person precisely in the left image from a stereo camera. The adaptive kernel function was combined with the regional feature of person and the Epanechnikov function, which can reduce the effect of the background pixel on the target’s color probability distribution. Compared with the traditional Mean Shift algorithm, the presented algorithm can track the target successfully when the background has the same color. Furthermore, the searching area is narrowed by the RFID, so that the computational cost is reduced. The average computing time is 62.11 ms/frame, which satisfies the requirements of real-time target tracking. The experimental results indicate that the proposed tracking method can complete the target tracking in a background with the same color, short-term occlusion, fast moving, and a sudden turn for a mobile robot.