Automatic detection of a designated building area (DBA) is a research hotspot in the field of target detection using remote sensing images. Target detection is urgently needed for tasks such as illegal building monitoring, dynamic land use monitoring, antiterrorism efforts, and military reconnaissance. The existing detection methods generally have low efficiency and poor detection accuracy due to the large size and complexity of remote sensing scenes. To address the problems of the current detection methods, this paper presents a DBA detection method that uses hierarchical structural constraints in remote sensing images. Our method was conducted in two main stages. (1) During keypoint generation, we proposed a screening method based on structural pattern descriptors. The local pattern feature of the initial keypoints was described by a multilevel local pattern histogram (MLPH) feature; then, we used one-class support vector machine (OC-SVM) merely to screen those building attribute keypoints. (2) To match the screened keypoints, we proposed a reliable DBA detection method based on matching the local structural similarities of the screened keypoints. We achieved precise keypoint matching by calculating the similarities of the local skeletal structures in the neighboring areas around the roughly matched keypoints to achieve DBA detection. We tested the proposed method on building area sets of different types and at different time phases. The experimental results show that the proposed method is both highly accurate and computationally efficient.