Aiming at the problems of low detection accuracy and missed detection caused by complex contour information, large change of shape and small size contraband in X-ray images, an improved GELAN-YOLOv8 model based on YOLOv8 is proposed. First, the RepNCSPELAN module based on generalized efficient layer aggregation network (GELAN) is introduced to improve the feature extract ability for contraband. Second, the GELAN-RD module is proposed by combining deformable convolution v3 (DCNv3) and RepNCSPELAN module to adapt contraband with different postures and serious changes in size and angle. Third, the spatial pyramid pooling is improved, so that the model can pay more attention to the feature information of small target contraband. Finally, the Inner-ShapeIoU is proposed by combining inner-intersection over union (Inner-IoU) and Shape-IoU to reduce the false detection and missed detection and speed up the convergence of the model. Results on the SIXray dataset show that the mAP@0.5 of the improved algorithm are 2.8 percentage points higher than YOLOv8n, and the performance is better than YOLOv8s. The GELAN-YOLOv8 effectively realizes the real-time detection of contraband in X-ray images.