A multiscale feature fusion-based traffic sign detection (TSD) algorithm is proposed to address the problem of poor performance of existing object detection algorithms in identifying small target traffic signs. First, a novel cascaded multiscale feature fusion network was designed, which fully utilizes the multiscale sequence feature fusion structure and triple feature encoding module, enabling the network to better integrate the detailed and global features of traffic signs. Second, deformable attention mechanisms were incorporated into the backbone network to enable the model to focus on relevant regions and capture richer image features. Finally, the use of the inner intersection over union (IoU) loss function improved the generalization performance of the TSD-YOLO model. The experimental results on the CSUST Chinese traffic sign detection benchmark (CCTSDB) dataset show that the average accuracy of the improved model was 55.3%, which is 3.2% higher than that of YOLOv8s. In addition, the performance on the visual object classes (VOC) dataset and benchmark Tsinghua-Tencent 100K (TT100K) dataset highlights the excellent generalization performance of the model.