Fisheye cameras offer lower deployment costs than traditional cameras for detecting the same scene. However, accurately detecting distorted targets in fisheye images requires increased computational complexity. To address the challenge of achieving both accuracy and inference speed in fisheye image detection, we propose an enhanced YOLOv8m-based fisheye object detection model, which we refer to as Fisheye-YOLOv8. First, we introduce the Faster-EMA module, which integrates lightweight convolution and multiscale attention to reduce delay and complexity in feature extraction. Next, we design the RFA-BiFPN structure, incorporating a parameter-sharing mechanism to enhance the detection speed and accuracy through receptive field attention and a weighted bidirectional pyramid structure. In addition, the lightweight G-LHead detection head is introduced to minimize the number of model parameters and reduce complexity. Finally, the LAMP pruning algorithm is introduced to balance improvements in recognition accuracy with inference speed. Experimental results demonstrate that Fisheye-YOLOv8 achieves mean average precision values of 60.5% and 59.7% on the Fisheye8K and WoodScape datasets, respectively, which is an increase of 2.2 and 1.2 percentage points compared to YOLOv8m. Moreover, the proposed model's parameter and computational complexity are only 20.5% and 29.7% of those of YOLOv8m, respectively, with a detection speed of 118 frames/s. The proposed model meets real-time requirements and is better suited for fisheye camera deployment than the other models.