A novel, simple and efficient memory gradient covariance tracking algorithm is proposed, which can optimize the distance function between the covariance target mould and the candidate target to search the best matched target quickly and accurately. In order to overcome the low efficiency of the exhaustive local searching in steepest descent algorithm, the memory gradient algorithm is taken full advantages to avoid converging to local optimal point. To reduce the calculation burden of the similarity metric for high dimensional positive symmetric covariance matrices under Riemannian space, Jensen-Bregman LogDet (JBLD) divergence metric is utilized to measure the similarity of covariance features. Besides that, the JBLD metric contributes to fast computation of the gradient under the framework of the gradient optimization algorithm. In the experiment, multi-scenario video standard testing library and new evaluation indexes are used. The experiment results show that the performance of the algorithm is better than compared algorithms.