The concealed object detection in millimeter wave (MMW) image is of great significance in non-contact body inspection. At present, MMW radar has been able to obtain 3D images, which are simply compressed into 2D images in current methods in general. However, such a rough processing does not take the information along the depth direction into account which results in a bottleneck of detection accuracy. To address this issue, a novel framework for MMW image concealed object detection is proposed, in which a 3D image is regarded as a sequence of 2D cross-sectional images and the most of the internal logic relations of features in the crosss-sectional images can be explored along the sequential direction, i.e. the depth direction of the 3D image. The framework consists of a Convolutional Neural Network (CNN) and a Long Short-Term Memory (LSTM) network. The former is used to extract the multiscale features in each 2D cross-sectional image while the latter is used to explore the global correlation of the above features along the depth direction to achieve feature-level information fusion and improve the accuracy of 2D location prediction. Experimental results show that the proposed method achieves remarkable results comparing to the known detection method based on 2D MMW images.