Cracks pose one of the most safety hazard to concrete building structures. A lightweight crack segmentation algorithm with improved DeepLabV3+ is proposed for efficiently segmenting concrete cracks and assessing their hazards in a timely manner. Firstly, MobileNetV3 is used as the lightweight backbone to significantly reduce the number of model parameters. Secondly, the attention-based intrascale feature interaction (AIFI) module is used to model the global information, and the normalization-based attention module (NAM) is introduced to facilitate the interaction of multi-level crack feature information. In addition, the mixed model of both self-attention and convolution is introduced after extracting the low-level high-resolution features, which captures the detailed features more efficiently; and finally, the C2f-SCConv module is designed to decode the fused high- and low-level feature streams, reducing computational redundancy and improving the perception of multi-scale features. Experimental results on the public crack datasets Concrete3k and Asphalt3k show that the number of parameters of the proposed model is reduced by 88.1% compared with that of the DeepLabV3+ model, the pixel accuracy is improved by 0.02%, the mean intersection over union (mIoU) reaches 86.21%, and the average frame rate is 47.91 frames per second. It means that the proposed methods reduce complexity of the model while improve segmentation efficiency to the cracks significantly.