Cracks on the surface of rail sleepers may cause hidden dangers to rail transit. Aiming at the problems of poor universality, low accuracy and low recall of rail crack detection methods, a rail crack detection algorithm YOLOv5s-CBE based on improved YOLOv5s is proposed. Firstly, the CA attention module is added to the backbone C3 module and between C3 and SPPF respectively to capture the channel relationship and location information from the two dimensions of channel and space, so as to improve the feature extraction capability of YOLOv5s backbone network. Secondly, in the neck part of YOLOv5s, BiFPN is used to fuse different scale information to obtain the output feature map with rich semantic information; At the same time, the weighted bi-directional feature fusion pyramid structure adjusts the contribution of input feature maps of different scales to the output by introducing weights, optimizes the feature fusion effect, reduces the loss of feature information in the convolution process, and improves the detection accuracy. Finally, the loss function CIoU in the original yolov5s is changed to EIoU. EIoU not only considers the distance and aspect ratio of the center point, but also considers the real difference in width and height between the prediction frame and the real frame, which improves the prediction accuracy of the anchor frame. Compared with the original network YOLOv5s, the model size of YOLOv5s-CBE rail crack detection network on the self-made rail crack data set is reduced by 1.0 MB, the accuracy mAP is increased by 3.7%, the recall rate is increased from 73.5% to 76.2%, and the phenomenon of missing detection of cracks of different sizes is reduced. It has certain advantages and practical value.