To solve the problems of high misdetection rates and the low detection precision of far and small objects with current three-dimensional (3D) object detection algorithms, an improved 3D object detection algorithm based on PointRCNN is proposed. The improved algorithm adopts the spatial autocorrelation algorithm in the preprocessing stage to reduce the dimension of data, effectively removes irrelevant and noisy points, and optimizes the network's ability to extract features and identify key objects. This study also proposes a module called MGSA-PointNet to improve the point cloud encoding network of PointRCNN. The module takes advantage of the manifold self-attention mechanism to extract spatial information in the original point cloud more accurately. It incorporates the grouping self-attention mechanism to reduce the parameter counts in the self-attention weight coding layer while improving the efficiency and generalization ability of the model and enhancing the feature extraction ability of the network. Compared with PointRCNN on the KITTI dataset, the proposed algorithm enhances the accuracy of the 3D detection of cars and cyclists in complex scenes by 2.10 percentage points and 2.14 percentage points, respectively, and improves the average accuracy of 3D pedestrian detection by 5.21 percentage points, thus proving the effectiveness of the algorithm.