A vehicle detection algorithm based on deformable feature fusion is proposed to solve the problem of low vehicle detection accuracy in environment perception of unmanned driving system. We propose a 3D vehicle detection algorithm based on deformable feature fusion. Firstly, the convergence speed and performance were improved by the road scene enhancement algorithm. The ground point cloud is removed to reduce the interference of irrelevant point cloud. Then, a deformable feature fusion module was constructed to adaptively align the pose and position information between different modal data to improve the utilization efficiency of multi-modal data. The loss function was optimized, and the adversarial loss was added to judge the authenticity of vehicle motion, so as to improve the detection accuracy of the network for small targets. Finally, the best weight of the network model is obtained by training, and the KITTI data set is used for testing, which can achieve better vehicle recognition effect. The experimental results show that the average precision value is 83.26%, and the average detection time is 0.15 s. The algorithm can quickly and accurately identify the vehicle in the unmanned driving system.