The popularization of unmanned aerial vehicle (UAV) has brought great security risks to people. A UAV video detection algorithm based on the active subspace robust principal component analysis (ASRPCA) fused with the five-frame difference is designed to solve this problem. First, an alternating iteration method combined with the augmented Lagrange multiplier method is used to optimize and solve the ASRPCA model, thereby obtaining the background image of the current frame of the video sequence. Second, the background image replaces the intermediate frame of the five-frame difference. Finally, differential binarization operation is performed simultaneously on the intermediate frames, previous two frames, and subsequent two frames. This makes UAV have a better detection result and a denoising ability. The experimental results show that under different backgrounds, compared with the algorithm of total variation regularized RPCA (TVRPCA), the average of recall rate, precision rate, and comprehensive performance of the proposed algorithm is increased by 5 percent, 4.8 percent, and 5 percent, respectively. The running time is approximately 0.51 s per frame, which meets the offline real-time requirements of the target algorithms.