This paper focuses on multi-target tracking algorithm based on Random Finite Set (RFS).The Gaussian Mixture Probability Hypothesis Density (GM-PHD) filtering algorithm is improved,and an algorithm based on GM-PHD is proposed,which can be used in Infrared Search and Track (IRST) system for nonlinear bearing-only tracking.The algorithm makes Bayes estimation to state set and measurement set,and no data association is needed.Therefore,the algorithm overcomes the problem of the traditional data association based multi-target tracking algorithm that it is difficult to make data association under the conditions of strong clutter and unknown/changing target number.It can improve the tracking ability of the IRST while making real-time estimation to the number of targets.