In order to solve the problem of poor applicability and high false alarm rate of target detection algorithm in unmanned aerial vehicle (UAV) detection systems in different scenarios, a UAV detection system based on a field-programmable gate array(FPGA) and digital signal processor architecture was designed by using infrared small target detection algorithm which could be applied to different complex backgrounds. Firstly, a bilateral filter algorithm was used to smooth the background and preserve the edge of the target region. Then, an improved multi-scale top-hat algorithm was adopted to enhance the target and suppress the background to improve the contrast difference between the target and the surrounding area. Finally, the adaptive threshold segmentation method based on maximum and average values was used to extract the target. The experimental results show that the detection rate of the system is 98.15%, and the overall delay is 33.33 ms. Compared with the existing typical infrared small target detection algorithms, the signal-to-noise ratio gain and background suppression factor of this proposed algorithm are increased by 6.8 times and 7.44 times on average, respectively, which effectively suppresses the background and enhances the target. The algorithm can effectively solve the problem of infrared small target detection in complex backgrounds, and it is helpful to improve the applicable ability and detection ability of the UAV detection system in different scenarios.