Traditional sparse representation tracker use simple grayscale characteristics in calculating sparse coefficient, which is easily affected by the heavy occlusions and deformation. To this end, a local adaptive weighting algorithm is put forward to increase degree of differentiation between the candidate targets affected by shade, deformation, etc and not affected by the shade, deformation, etc. In addition, the general sparse representation algorithm use a small number of target templates to build a complete dictionary, which unable to get a better sparse coefficient. Inverse structure sparse representation algorithm, using the candidate target which contains rich target and background features to build a complete dictionary to reconstruct the target template under the condition of the same dimension target template better sparse coefficient can be obtained, is proposed. Experiments show that the proposed algorithm in the small differences between target and background or serious barrier, deformation, can better track the target.