For the large computation and the real-time tracking to be hard to achieve by traditional fragment based tracking algorithm, an improved fragment based algorithm was proposed. Firstly, in order to reduce the negative effect yielded by the background noise, the region of the tracked object was divided into more fragments. Then, the position and the range of the search region were identified according to the movement information of the object. By utilizing a fast hierarchical adaptive search approach which adopts different search patterns in different steps, most of the calculations for invalid positions were skipped, and the coordinate where the candidate was most similar with the object template was obtained quickly. Moreover, the improved algorithm was implemented and optimized on a DSP. Experimental results indicate that the improved algorithm can process the image of 768 pixel×576 pixel on DM642 at a processing speed of 30 frame/s. Compared with the traditional fragment based tracking algorithm,it shows a more precise tracking and saves the processing time about 47.5%. These results show that the improved algorithm overcome the shortcomes from traditional fragment based tracking algorithms, and can achieve a real-time tracking with better tracking performance.