With the widespread use of unmanned aerial vehicle (UAV) technology in military, civilian, and other fields, the demand for high-precision, low-power intelligent UAV tracking systems is also increasing. Aiming at the problems of scale variation, out-of-view, and occlusion in UAV tracking tasks, a real-time tracking algorithm for UAV based on light-weight Siamese network was proposed. Firstly, the lightweight convolutional neural network MobileNetV2, which is easy to be deployed in embedded devices, is selected as the feature extraction backbone network. Secondly, the channel spatial coordination attention module is designed to enhance the adaptive and discriminative ability of the model. Thirdly, the region proposal network is equipped, and the foreground background classification and boundary box regression response map are obtained through correlation. Finally, the weighted fusion multilayer response map is calculated and proposal region screening strategy is adjusted to obtain more accurate tracking results. Simulation experimental results on the UAV tracking dataset show that the tracking accuracy is improved by 3.5% compared to the current mainstream algorithm SiamRPN, and the algorithm can better cope with complex and changeable scenes. Meanwhile, on the NIVIDA RTX 2060 GPU, the tracking speed achieves 60 frame/s.