Given the large scale, uneven distribution, and large scale changes of small targets and the complex background in remote sensing images, an improved cascade algorithm SA-Cascade is proposed. This algorithm uses a recurrent feature pyramid network to strengthen the feature representation generated step by step, thereby improving the detection rate of small targets. The region proposal generation network based on the learnable anchor is utilized to locate the remote sensing target accurately. The feature adaptation module and feature fusion module are introduced to improve the performance in detecting images with complex backgrounds. On the basis of the cascade algorithm, the two-branch detection head is adopted to improve the performance of the model for detecting small targets. A comparative experiment of various algorithms is performed on TGRS-HRRSD-Dataset and VisDrone-DET dataset. The experimental results show that the improved cascade algorithm can detect and locate remote sensing image targets more accurately. Compared with the original cascade algorithm, the improved one increases the accuracy on the two datasets by 2.94% and 9.71%, respectively.