Object tracking for the unmanned aerial vehicle (UAV) in videos is an important part of the Anti-UAV task. The complex background during low-altitude flight and the small imaging size are two difficulties for UAV object tracking. A Siamese neural network object tracking algorithm (SiamAU) is proposed, which is based on SiamRPN++ in combination with an improved backbone and a feature rearrangement technique. Firstly, ECA-Net attention module is integrated into the backbone network, while the activation function is improved to enhance the representation ability of convolution features in complex background. Then, channel number of the last three convolution features is rearranged in order to make full use of low-level features that are conducive for small object tracking. The rearranged feathers are further fused to obtain the improved feature map. Finally, On the DUT Anti-UAV dataset, SiamAU algorithm achieves success and precession scores of 60.5% and 88.1%, an improvement of 5.6% and 8.1% in comparison with the baseline algorithm. Extensive experimental results on two public datasets validate that the proposed SiamAU achieves better UAV tracking performance and outperforms previous methods, especially in small object and complex background scenarios.