Aiming at the problem of high false alarm rate of infrared small target detection of unmanned aerial vehicle remote sensing system in complex environment, a two-stage infrared small target detection model of the unmanned aerial vehicle remote sensing system is proposed with combination of convolutional neural networks. In the first phase, the Unet neural network is taken advantage to learn the deep semantic features and shallow location features of targets in the infrared image, meanwhile, the infrared week and small target signal is enhanced and the background signal is suppressed. In the second phase, Faster R-CNN is utilized to analyze the output image of the first phase, to detect the location and bounding box of infrared small target. Validation experiment is carried on the public infrared small target detection dataset of the unmanned aerial vehicle remote sensing system, the results show that the detection precision of the proposed model for infrared small target increases by 13.2、9.8 and 13 percentage points on three complex background datasets respectively, and the processed frames per second increase 11、14 and 13.