Detecting the concealed object from millimeter wave images is one of the key techniques to construct an intelligent millimeter wave based security inspection. To address the issue that the concealed objects are inspected hardly due to their locality and low identifiability in the millimeter wave images, a dynamic self-attentive bilinear convolutional neural network (DSA-BCNN) is proposed to train with image-level labels to detect the concealed objects. Self-attention mechanism is utilized to guide network to extract the features from concealed object regions, which enhances the network capability to depict the global information. Simultaneously, the second order features are constructed by bilinear pooling to enrich the representation of subtle differences between concealed objects and non-detected regions. Experimental results verify the propose method effectiveness, which is superior than others in terms of each evaluation metric, and the accuracy is 93.6%.