With the frequent appearance of UAVs in several recent local wars and armed conflicts, the study of UAV detection and tracking technology has become a research hotspot in imagery and other fields. Due to the characteristics of low altitude UAV targets such as large mobility, small size, low contrast and complex background, their capture and tracking is a major challenge in the field of photoelectric detection. To address these difficulties, this paper proposes a real-time long tracking method based on YOLOv5 and CSRT algorithm optimization to achieve stable tracking of UAVs in clear sky, urban and forest scenes. First, two capture networks with different resolutions are established for different stages of tracking, and the two networks are optimized for small target detection and performance optimization respectively, and positive and negative samples are added to the UAV data set according to its characteristics to achieve data enhancement. Then, the CSRT algorithm is optimized using GPU and combined with feature point extraction to construct a low-altitude UAV detection and tracking model. Finally, the algorithm is deployed using Tensorrt and experimented on a self-built dataset. The experimental results show that the proposed method achieves a tracking performance of 400FPS on RTX 2080Ti and 70FPS on NVIDIA Jetson NX. Stable long-time tracking is also achieved in real field experiments.