This study proposed a method for optimizing and regularizing building contours step by step to investigate the irregular problem of building contours extracted by classification in a high-resolution remote sensing image. The initial building contours were extracted and reconstructed by polygon fitting based on the initial building results extracted by the image classification and verification. The best fitting circumscribed rectangles consistent with the building axis inclination were then obtained. Subsequently, the contour lines of the building polygon and their corresponding best circumscribed rectangular boundaries were divided into equal segments. After which, the Hausdorff distances between the building polygon and rectangular boundary segments were calculated. Contour preliminary regularization and optimization were accomplished herein by replacing the building segments, whose Hausdorff distances were smaller and satisfied with the replacement condition, with the corresponding best-fitting circumscribed boundary segments. Deep optimization for a complex partial contour not well optimized in the former step was also explored. The feature corner points on the complex boundary were extracted, matched, sorted, and removed to keep the best ones based on the Shi--Tomasi algorithm. Lastly, the remaining points were connected and reconstructed to optimize the complex local contour. As a result, the edge expression degree accuracy and the extraction accuracy after the contour optimization were improved. The experimental comparison and analysis results of multiple remote sensing images show that this method is not only suitable for the contour optimization of building results extracted by different classification methods but also effectively improves the edge expression accuracy of building contours. According to the change of complex building contour details, the proposed hierarchical optimization method was more accurately adaptive than the reference contour optimization method. It also achieved a better overall optimization accuracy. In other words, the building edge accuracy and regularity can be effectively improved, and the true building shape can be more accurately reflected.