In view of the low accuracy of the current bridge disease detection algorithm based on convolutional neural network, an improved YOLOX algorithm was proposed to improve the detection accuracy. By using the feature information of the shallow layer of the backbone network, the feature extraction enhancement network was improved, and the feature information of the same layer was added for fusion. An improved coordinate attention mechanism was introduced to combine the position information and the channel information to enhance the network recognition of bridge diseases. At the same time, the localization loss function was improved. The experimental results show that the accuracy of the improved YOLOX network structure for bridge disease detection reaches 92.11%, which is 4.40% higher than the original network.