Building extraction from remote sensing images is of great significance to the construction of smart cities. Aiming to improve the low accuracy of traditional methods in extracting remote sensing images with a complex background, a remote sensing image building extraction method (MA-Unet) based on U-Net is proposed. This method mainly uses an encoder and a decoder. A convolutional block attention module is introduced into the encoder, in which a channel attention module is used to screen more important features and suppress invalid features, and a spatial attention module is used to screen deeper semantic features. An atrous spatial pyramid pooling module is introduced to extract features with different scales. In the decoder, to fuse object features with different scales, feature maps in the decoder are upsampled and connected in series. This information aggregation solves the difficulty of detecting objects with different scales to some extent. The experimental results show that MA-Unet method is superior to the U-Net method in terms of accuracy, precision, and intersection over union (IoU) by 1.7 percentage points, 2.1 percentage points, and 1.6 percentage points on the Massachusetts building dataset and by 1.1 percentage points, 1.4 percentage points, and 2.3 percentage points on the WHU building dataset, respectively. It is a more effective and practical target extraction method.