A moving object detection algorithm based on new background extraction is proposed to rapidly and efficiently detect moving objects in various environments. First, N consecutive images are read from the video. For any pixel position, the corresponding positions of each frame image and other images are subtracted to obtain N groups of difference sequences containing N differences. Next, based on the rectangular radial basis function, the number of differences within the width of the rectangle in each difference sequence is counted. Finally, the pixel value corresponding to the maximum frequency difference sequence is used as the background and the moving target is extracted via background subtraction. The experimental results show that under the condition of a specific amount of data, the structural similarity value of the background established by the proposed method and the real background is 0.162 higher than that of the ViBe algorithm. The precision, recall, F1 measure, and false positive rate indexes of the moving target detection results are better than those of the ViBe and GMM algorithms. Therefore, the proposed algorithm is a moving target detection algorithm with high accuracy and anti-interference ability.