For the task of weak-feature target tracking in infrared image sequences, existing methods failed to track the target accurately for lack of information in the target′s surface and the interferences from the strong noise in the scene. Focusing on this problem, a particle filter method is proposed by the Spatial-Temporal Orientation Energy (SOE). The method describes the target by using the SOE histogram. The similarity of the SOE histograms between the particles and the target is employed as the observation input to the particle filter. It is noted that the method is high robustness to target illuminated radiation as the SOE feature reflects target dynamics. Experimental results show that the proposed method has good stability to track targets in different scenes and is more effective and adaptive than the traditional grey-scale feature.