In consideration of the effect of stability of star tracking processing on its overall performance, how to track the stars in the current visual field according to the star information identified in the previous moment is discussed. To obtain a higher stability in star tracking, the advantages and disadvantages of existing star tracking algorithms are analyzed, and the method of "sorting before bidirectional selective rule out matching" is proposed. The first step selects the effective reference star on the basis of determining the marginal area of the FOV(Field of View), which effectively reduces the probability of mismatching and ensures the validity of matching. Following that it uses bidirectional recurrence idea to obtain the successful matching capacity that “there are more observation stars within the neighborhood of the reference star”, so as to increase the number of stars successfully tracked in each frame of rapid star tracking. A tracking experiment was performed in a outdoor field for a star sensor. The comparison experiments on start tracking in 100 directions generated by Monte Carlo method show that when the attitude movement has a very large angular velocity, the proposed method can achieve about 91.44％ fast star tracking. However, it just is 77.18％ by other traditional methods. The proposed method has a significance to improve the overall performance of the star sensor.