When extracting foreground targets in the complex background with dynamic interference factors, the existing algorithms of extracting foreground targets in visual background are prone to ghost image and false detection, so an improved algorithm based on visual background is proposed in this paper. First, according to the time series and position characteristics of pixels, the matching probability, matching degree, and brightness information of the pixels are calculated. Second, background model matching the current complex background is updated in real time, and the background model is initialized. Finally, the video in various complex backgrounds in the CDnet 2014 dataset is tested, and compared with the classic Gaussian mixture model, visual background extraction (ViBe) algorithm, and improved ViBe algorithm. Experimental results show that the algorithm can efficiently remove the effects of ghosts in various complex backgrounds, had a high extraction precision, which greatly reduces the misclassification rate and missed detection rate of the extraction results, and improves the efficiency and robustness of the algorithm in complex background.