The difference in projection angle leads to changes in the shape and size of objects, which limits the improvement of the detection performance of small objects by the two-dimensional (2D) concealed object detection method based on projected views of active millimeter wave (AMMW) holographic images. For this reason, a three-dimensional (3D) concealed object detection method based on point clouds was proposed for the first time. AMMW holographic images were converted into point clouds by thresholding, and then were input into the 3D object detector SECOND, which was improved by dilated convolution and multi-branch structure, to extract the 3D geometric understanding of the objects and their multi-scale context information to improve the detection ability for small objects. The experimental results showed that compared with the projection-based 2D detection methods, the average recall (AR) of this method was improved by 3.33%, which proved the effective improvement of localization accuracy. The detection rate and the average precision (AP) was relatively improved by 8.75% and 7.11%, and the false alarm was reduced by 1.78% at an intersection over union (IOU) threshold of 0.5. The average AP under different IOU thresholds was improved by 4.30%. The detection accuracy was effectively improved. The detection speed was 17.3 FPS, which reached the real-time level.