As tracking algorithms based on Intensity Variation Function(IVF) can track effectively rigid targets in Forward Look Infrared(FLIR) imagery, but can not satisfy the need of a pedestrian tracking for robustness, a novel pedestrian tracking algorithm based on modified IVF was proposed. The necessity of describing the thermal signatures of pedestrians with multiple hot spots in divided subregions was analyzed. Then, the hot spots were detected in a target window from frame to frame by the modified IVF and an adaptive update mechanism for a target window was established to solve the scale change. Finally, the motion feature descriptors based on hot spots were used to remove the outliers detected unaccurately in the background. Comparative experiments on challenging thermal scenes demonstrate that the proposed algorithm outperforms the state-of-the-art approaches in real-time performance by removing the template matching step of original algorithm and decreasing the matrix dimension of searching objects. Moreover, by better robustness against many visual tracking algorithms with the multiple hot spot strategy, it is suitable for the pedestrian tracking in FLIR imagery with the interference of occlusion, scale changed and lower contrast.