To address the poor performance of building extraction caused by low discrimination between the building target and background environment in remote sensing images, a high-order statistics integrated encoder-decoder network method was proposed to improve the accuracy of automatic building extraction. First, the deep encoder-decoder network was used to extract the low-order semantic features of building targets. Then, the polynomial kernels were used to achieve the high-order description of intermediate feature maps to improve the ability to recognize ambiguous features. Finally, the lower-order feature maps cascading with the higher-order features were sent to the end of the network to obtain the segmentation results of the building. Experiments on the Massachusetts Buildings dataset show that the proposed approach can achieve recall of 85.1%, precision of 77.5% and F1-score of 80.9%. Compared with the baseline network, the proposed approach is 4% higher in the metric of F1-score. The proposed method improves the performance of encoder-decoder networks for automatic building extraction of remote sensing images, and can extract building targets with low discrimination more accurately; hence, it has a good application value.