The visual background extraction (Vibe) algorithm cannot effectively remove the shadow of the target, and cannot quickly remove the ghost phenomenon. To address the shortcomings of Vibe, an improved YUV_Vibe fusion algorithm is proposed. The algorithm expands the value range of the sample field, which effectively avoids the repetitive selection of the same samples. The updating factor is adjusted from 16 to 4, and the number of sample updates is set at 2, which accelerates the update rate of the background to eliminate the rate of ghost detection. The fusion of the YUV color information features with the Vibe algorithm eliminates the influence of shadows. By constructing a double fusion model, the false detection rate of shadows is effectively reduced. The algorithm is experimentally applied to video datasets. The test results reveal that the improved YUV_Vibe fusion algorithm has improved the accuracy and recognition rate, and the experimental detection results are more accurate.