To address the issues of missed and false detection in high-resolution optical remote sensing images caused by complex backgrounds, large variations in target sizes, and a high proportion of small targets, an improved YOLOv8s object detection algorithm is proposed. First, the weighted bi-directional feature pyramid network (BiFPN) structure is introduced to replace the original neck network, and an additional branch from the backbone network is added to capture shallow features, thereby enhancing the performance of small-target detection. Second, the efficient multi-scale attention (EMA) mechanism is incorporated into the Bottleneck of the neck's cross-stage partial network fusion (C2f) module to improve the focus on spatial and channel information while reducing interference from irrelevant information. Finally, based on SCYLLA-IoU (SIoU), SWIoU loss function is designed by combining the dynamic non-monotonic focus coefficient from wise-IoU (WIoU). This strengthens the performance of bounding box regression and considers the directional matching between the predicted and ground truth boxes, further improving the detection accuracy of the algorithm. On the DOTA dataset, the improved algorithm achieves a mean average precision (mAP50) of 73.0%, which is a 3.0 percentage points increase compared to that of YOLOv8s, and its detection results also outperforms other mainstream algorithms. Furthermore, the experimental results obtained on the RSOD and NWPU NHR-10 datasets further validate the generality and adaptability of the proposed algorithm across different datasets.