The contours of buildings extracted using the image classification method are commonly irregular and involve serration issues that are primarily caused by incorrect recognition. Therefore, this paper proposes a building contour optimization method that combines Hausdorff distance and a suitable circumscribed rectangle that conforms to the contour and axial direction of buildings. Firstly, initial results of buildings are extracted using the shifted shadow segmentation and classification principle. For each building, a corresponding fitting polygon is acquired by applying a fitting principle to its building edge. Subsequently, the building axis is assessed using the minimum circumscribed rectangle of the building polygon-fitting result. Based on the axis, a suitable circumscribed rectangle is selected. Furthermore, the building contour and its suitable circumscribed rectangle are respectively divided into equal segments. Meanwhile, the Hausdorff distance between two kinds of segmentations is calculated. If the distance satisfies the substitution rules, building contour segments are replaced with circumscribed rectangular edge segmentation to optimize building regularization. Thus, the proposed method helps in improving the accuracy of building boundary and in promoting building extraction precision. It is tested on several remote sensing images. Compared with other building extraction methods, the results show that the overall accuracy of the proposed method is better than that of the other two reference methods. Moreover, the accuracy and regularization of building contours and the overall precision of building extraction results have been effectively improved. As a result, building shape is more accurately reflected.