To address the problem of low accuracy of building extraction in high-resolution remote sensing images due to the diverse shapes and sizes of buildings and large number of parameters in traditional segmentation models， a Lightweight Multi-scale Difference network （LMD-Net） based on encoding-decoding is proposed. First， to avoid the invalid parameters caused by the degraded model performance due to the stacking of single feature processing units， a lightweight differential model is designed to improve the performance by integrating the functional differences of codec structures. Next， a Multi-scale Dilation Perception （MSDP） module is introduced to enhance the ability of the network to capture multi-scale target features. Finally， the double fusion mechanism is used to effectively aggregate the feature information of the deep jump connection and deep decoder to enhance the feature recovery ability of the decoder. To verify the validity and applicability of LMD-Net， the open source WHU building dataset was used as the data source to evaluate the accuracy and efficiency of LMD-Net and the common semantic segmentation network as well as the results from recent relevant literature. The results show that LMD-Net has obvious advantages in both efficiency and accuracy， which not only greatly reduces the parameter number and calculation amount of the model but also improves the intersection ratio and accuracy by 3.23% and 2.57%， respectively. Consequently， this model is advantageous in the field of building extraction based on high-resolution remote sensing images to generate an urban spatial information base.