Aiming at the problem of intricate road shape and structure in high-resolution remote sensing images, where narrow and small roads are extracted incorrectly or omitted, a lightweight remote sensing image road extraction method based on Atrous Space Pyramid Pooling and Attention Mechanism is proposed. Firstly, based on the original HRNet network, multi-scale road information fusion is realized by introducing the ASPP. Secondly, the Squeeze and Excitation channel attention mechanism (SE-networks) is introduced to enhance the quality of network feature representation. Finally, using deep separable convolution to improve the network residual module to realize the model lightweight and reduce the complexity of model calculation. Experimental results on the publicly available dataset show that the accuracy, precision, recall, F1 score and the MIoU of the proposed algorithm was improved respectively by 5.35%, 2.15%, 4.1%, 3.15% and 14.34%, compared with the original HRNet network, and reduce the number of parameters by 35.6%. Compared with other networks, the algorithm highlights the characteristics of small roads, and the prediction results have good continuity and integrity. As the small size, the proposed model is easier to deploy in real-time detection equipment. The proposed model effectively reduces the road extraction fault and missing, implements a stronger adaptability, higher segmentation accuracy, more lightweight multi-scale road semantic segmentation algorithm.