In the detection of moving objects in complex dynamic background, there are many problems, such as incomplete extraction of foreground objects and false detection of dynamic background as foreground. To solve the above issues, a moving object detection model that combines a nonconvex rank approximation function and a three-dimensional total variation (3D-TV) regularization term is proposed. Based on the original robust principal component analysis model, the proposed model introduces a nonconvex rank approximation function to describe the low rank of the video background part and uses the 3D-TV regularization term to constrain the foreground part in time and space. Finally, the alternating direction multiplier method is used to solve the proposed model. Furthermore, the experimental results show that the model can effectively improve the accuracy of moving target detection when dealing with complex scenes, such as dynamic background and bad weather, and has a better visual effect than existing methods.