Accurately identifying building distribution from remote sensing images with complex background information is challenging. The emergence of diffusion models has prompted the innovative idea of employing the reverse denoising process to distill building distribution from these complex backgrounds. Building on this concept, we propose a novel framework, building extraction diffusion model (BEDiff), which meticulously refines the extraction of building footprints from remote sensing images in a stepwise fashion. Our approach begins with the design of booster guidance, a mechanism that extracts structural and semantic features from remote sensing images to serve as priors, thereby providing targeted guidance for the diffusion process. Additionally, we introduce a cross-feature fusion module (CFM) that bridges the semantic gap between different types of features, facilitating the integration of the attributes extracted by booster guidance into the diffusion process more effectively. Our proposed BEDiff marks the first application of diffusion models to the task of building extraction. Empirical evidence from extensive experiments on the Beijing building dataset demonstrates the superior performance of BEDiff, affirming its effectiveness and potential for enhancing the accuracy of building extraction in complex urban landscapes.