This paper presents an improved algorithm to solve the problem of ghosting in the visual background extractor (ViBe) algorithm for pedestrian detection. The initialization strategy of the ViBe algorithm is improved using the YOLO v3-spatial pyramid pooling (SPP) algorithm to eliminate ghosts. Thus, the YOLO v3-SPP algorithm detects pedestrians in the first frame, eliminates the detected pedestrians, and replaces the first frame of the ViBe algorithm with the output image to eliminate ghosts. The results of the analysis and verification show that the algorithm can effectively solve the ghost problem.