Crack detection is a key engineering task. Aiming at the problems of large number of parameters, large amount of calculation and weak real-time performance of the existing mainstream crack semantic segmentation models, an improved real-time segmentation algorithm of crack images based on fast segmentation convolution neural network (Fast-SCNN) was proposed. First, the spatial pyramid pooling (SPP) module with disadvantages of loss of pixel position information and large amount of calculation was optimized on the basis of Fast-SCNN. Then, the up-sampling method was improved to fully consider the relationship between pixels, and a lightweight positional self-attention module was proposed for up-sampling to improve the detection accuracy. Finally, the respective outputs of the dual branches highlight the crack-related regions and suppress the irrelevant backgrounds through the attention gates. The proposed algorithm can provide a more accurate pixel-level attention for the model, and can more effectively identify small cracks as well as improve the robustness of crack segmentation in complex backgrounds. Experiments show that, compared with the existing mainstream models and other lightweight models, the proposed algorithm further balances the segmentation accuracy and detection speed, and achieves an average intersection ratio of 80.31% and an F1 score of 76.74% on the crack dataset. The parameter amount is 1.20 M, the calculation amount is less than 1 G, and the inference speed reaches 151 f/s, which has high application value for the real-time segmentation and detection task of crack images.