To solve the problem of multiple targets’ detection and tracking under the complex environment, in this paper, an improved moving objects detection method is proposed based on four inter-frame differential method and optical flow algorithm. Firstly, four inter-frame difference method is used to process the of video sequences. Then objects in the video is detected accurately by the optical flow algorithm used on light streaming video sequences. This improved method enhances the processing speed of optical flow method and reduces the effects of environment’s illumination. Finally, the paper compares the proposed algorithm with particle filter, ViBe algorithm under different scenarios with different moving targets and individual number. This improved method is proved not only with good robustness, but also can work more quickly and accurately on the target detection and tracking.