To address the challenge of handling large volumes of point cloud data for three-dimensional (3D) object detection and the limited effectiveness in detecting small objects, in this study, an enhanced 3D target detection method is proposed that improves the YOLOv5 network based on the idea of Complex-YOLO algorithm. The proposed approach first tackles the issue of lengthy processing times due to extensive point cloud data by adopting the Complex-YOLO strategy of converting point cloud data into an RGB-Map format, which is more manageable for the YOLOv5 network. Enhancements to YOLOv5 include an angle prediction branch and a rotation frame regression loss function to accurately position rotating targets within the RGB-Map. Additionally, the YOLOv5 architecture is modified to better detect small objects by incorporating a feature fusion layer and a dedicated prediction head, which heightens the network's sensitivity to smaller targets. Furthermore, the convolutional block attention module (CBAM) attention mechanism is integrated into the network's neck to further enhance detection sensitivity. Experimental evaluations on the KITTI dataset confirm the superiority of the modified YOLOv5 method over the original Complex-YOLO, with improvements in mean average precision (mAP): Car type mAP increased by 7.48 percentage points, Pedestrian type by 12.54 percentage points, Cyclist type by 1.2 percentage points, and an overall increase of 7.08 percentage points across all categories, demonstrating the effectiveness of this algorithm.