In order to improve the detection capability of small target defects in steel surface inspection, an improved YOLOv8-SOE model is proposed. The model processes the P2 layer features by designing the FSCConv module. By compressing the P2 layer features and deeply fusing them with the P3 layer features, the model's sensitivity to small target features is effectively enhanced, while avoiding the computational burden caused by the introduction of additional detection layers. In order to further optimize the multi-scale feature fusion capability, cross stage partial omni-kernel (CSP-OK) module is used to optimize the multi-scale feature fusion, which improves the integration efficiency of features of different scales. The SIoU loss function is introduced to optimize the bounding box regression, which further improves the positioning accuracy. Experimental results show that the mAP of the YOLOv8-SOE model on the NEU-DET dataset achieves 80.7%, which is 5.4% higher than the baseline model, and has good generalization ability on the VOC2012 dataset. While improving the accuracy of small target detection, the model maintains a high computational efficiency and has good application prospects.