Aiming at the problem that modulation format recognition accuracy is susceptible to factors such as channel environment and background noise interference in visible light communication signal transmission, this paper proposes an improved YOLOv5s(You Only Look Once) algorithm for modulation format recognition of visible light communication signals. Firstly, the Mixup data augmentation method is introduced at the input end of the YOLOv5s algorithm network, and it is combined with the Mosaic data augmentation method in the original network to enhance the robustness of the network and improve the generalization ability of the algorithm among different modulation format signals. Secondly, adaptively spatial feature fusion (ASFF) is introduced into the Neck network to fully extract features from different levels and improve detection accuracy. The experimental results indicate that under mixed signal-to-noise ratio conditions, the mean average precision(mAP) of the proposed improved algorithm reaches 0.903, representing a 0.7% improvement compared to the original YOLOv5s algorithm. Furthermore, the mAP reaches a high of 0.993 when the signal-to-noise ratio is 20 dB.