To solve the problem in which a traditional ResNet101 model cannot effectively describe the detailed features of remote sensing images, leading to the unclear segmentation boundary between roads, trees, and buildings in complex scenes, a Multiscale-feature Fusion Dilated Convolution ResNet (MFDC-ResNet) was proposed. First, to obtain large-scale building feature information of remote sensing images, a dilated convolution was introduced in the deep residual network to capture richer multi-scale details. Second, to enhance the expression ability of the center point of dilated convolution on the building of feature images, a 3×3 convolution kernel was proposed to extract features in the local area of remote sensing images. Finally, a spatial pyramid pool model of multi-scale feature fusion was proposed to fuse the multi-scale features, obtain the building contextual information of different scales of remote sensing images, and complete the accurate segmentation of buildings. The results of the experiments show that the mean Intersection over Union (mIoU) of building segmentation in WHU is 0.820 and the recall rate is 0.882. The developed method can effectively overcome the influence of roads, trees, and other factors. Moreover, the building boundary can be extracted clearly and smoothly from the remote sensing images and the segmentation accuracy is improved.