ing at the problems of ghost and the noise interference from dynamic background in classic visual background extraction algorithm, an improved visual background extraction algorithm is proposed. The important feature information of pixels in complex environment can be collected by creating the auxiliary sample set. By introducing analysis of the pixel ghost factor and the region complexity, the matching threshold and updating rate of each pixel can be adaptively adjusted. With the pixel flicker analysis based on sliding window, the points which may be misdetected as foreground can be updated to the auxiliary samples according to probability. The comparative experiments in the multi-scene show that the proposed method can reduce the wrong classifications rate to as low as 1.49%, eliminate the ghost quickly, suppress the noise interference from the dynamic background, and ensure the complete recognition of foreground target. The results of the algorithm are more accurate in the complex environment.