To address the threat of small drones "black flying" to the public domain. Based on the multimodal image information of an unmanned aerial vehicle (UAV) target, a lightweight multimodal adaptive fusion Siamese network is proposed in this paper. To design a new adaptive fusion strategy, this module assigns different modal weights by defining two model training parameters to achieve adaptive fusion. The structure is reconstructed on the basis of a Ghost PAN, and a pyramid fusion structure more suitable for UAV target detection is constructed. The results of ablation experiments show that each module of the algorithm in this study can improve the detection accuracy of the UAV targets. Multi-algorithm comparison experiments demonstrated the robustness of the algorithm. The mAP increased by 9% when the detection time was basically unchanged.