Detection of concrete bridge surface damage is crucial for bridge maintenance. However, existing machine vision-based methods suffer from low detection efficiency and accuracy when dealing with small-scale damages and complex backgrounds. In this paper, a novel detection network based on YOLOv5 is proposed. By optimizing the YOLOv5 backbone network and introducing a global attention mechanism along with a multiscale pyramid spatial pooling structure, detection accuracy and efficiency are effectively improved, especially in the detection of small-scale damages in complex backgrounds. Experimental results show that average detection accuracy of the improved model increased by 4.1% compared to that of the original network structure, surpassing that of YOLOv7 and YOLOv5 in detection performance on small-scale damages, such as holes and complex backgrounds. The proposed method achieved a 2.3% increase in average detection accuracy and a 30% improvement in detection speed over YOLOv7.