In the visible and infrared scenes with complex background, such as rain and snow weather, leaf swaying, shimmering water, etc., fast and accurate extraction of a complete target has always been the primary problem in moving target detection. In order to be real time and aiming at the problems of existing video foreground extraction algorithms, such as dependence on prior information, low recall rate, lack of texture and large noise, a background modeling method based on histogram statistics and improved LBP (Local Binary Pattern) texture features is proposed. Firstly, the mode of each pixel histogram is used as the reference background without prior knowledge, which saves a lot of storage space. Then, an improved S_MBLBP texture histogram is proposed to model the background with the reference background by using neighborhood compensation strategy, which eliminates the most dynamic background and illumination changes, and realizes the accurate extraction of the target. Experimental results show that the proposed algorithm can quickly extract foreground targets in a variety of complex infrared and visible scenes, and can improve the accuracy and recall rate at the same time.