With the development of computer technology, the target tracking methods based on deep learning have become an important research direction in the field of computer vision. However, the target tracking methods still face great challenges in complex environment such as background interference and color proximity. Compared with the traditional color images, the hyperspectral images contain rich radiation, spatial and spectral information, which can effectively improve the accuracy of target tracking. A method was proposed in combination with attention mechanism and additive angular margin loss (AAML) to perform target tracking for hyperspectral images. The features of different combinations of bands were extracted by fused multi-domain neural networks, and then the fused attention mechanism model was designed to make the similar features from different combinations of bands integrated and strengthened. Therefore, when the target background color was similar, the network would pay more attention to the target object, which made the tracking results more accurate. On this basis, in order to make the distinction between target and background more discriminative, the AAML was adopted as loss fuction to effectively reduce the intra-class distance of similar samples, increase the inter-class distance between centers of positive and negative samples, and improve the network accuracy and stability during the training process. The experimental results show that the accuracy and success rate of the two tracking accuracy evaluation indexes can be improved by 1.3% and 0.3% respectively, which has more advantages than other methods.