The traditional building extraction methods for classification and identification are susceptible to interference from the complex environment and can lead to false detection. As a result, the building contour is irregular and inconsistent with the actual shape. To solve this problem, a building contour optimization method based on the main direction is proposed. Firstly, according to the image and building features, the initial extraction results of a building are pretreated to remove non-building patterns and scattered line segments. Then, the main direction of the building is determined by considering its positive circumscribed rectangle, and it is used to reconstruct the building contour for the preliminary regularization result. Finally, the boundary characteristic strength is calculated with the initial result to further reduce the deviation after regularization, and achieve depth optimization of building contours. In addition, by comparing with other methods, the relevant parameters are discussed and analyzed. The result indicates that compared with other methods, this method can effectively improve the integrity and regularity of the initial result, and it can finely optimize the buildings with different sizes and complexity to obtain higher optimization accuracy. The proposed method can be used as a reference method for optimization after building extraction.