In the rail surface defect detection, the rail surface image has the problem of uneven background, large variation of defect scale, and insufficient sample data. Therefore, this paper proposes a rail surface defect detection method based on image enhancement and improved Cascade R-CNN. First, the improved Retinex algorithm is used to process the rail surface image to enhance the contrast between the defects and the background. Then, an improved Cascade R-CNN is adopted to detect rail surface defects, and the intersection over union (IoU) balanced sampling, region of interest align and complete intersection over union (CIoU) loss are applied to solve the imbalance between training sample IoU distribution and the difficult sample IoU distribution, the misalignment between region of interest and extracted feature map caused by rounding quantization in region of interest pooling, and the inaccuracy of the regression loss Smooth L1 for the regression of predicted bounding box. Finally, the dataset of rail surface defect images is expanded using methods such as flipping transformation, random cropping, brightness transformation, and generative adversarial networks, so as to solve the phenomenon of over-fitting of network training caused by insufficient sample data. Experimental results show that the average accuracy of the proposed method, using ResNet-50 as the feature extractor, can reach 98.75%, which is 2.52% higher than the unimproved Cascade R-CNN, and the detection time is reduced by 24.2 ms.