A novel method for detecting moving objects from video sequences is proposed based on Markov random field. In order to overcome the drawback of subjective fixed threshold of traditional temporal segmentation, the difference image is modeled by Markov random field. A novel method for deciding the model size and initial parameters of MRF, and a new iteration method for greatly accelerating the convergence are proposed. Then, the Expectation-Maximization (EM) algorithm is carried out to obtain the Gaussian parameters and temporal moving area is detected. The temporal segmentation is then amended by morphological operations. Considering the lack of traditional spatial segmentation algorithm of watershed, an improved watershed algorithm in accord with the human vision characteristics, which can restrain over-segmentation effectively, is proposed. The temporal and spatial information fusion is fulfilled by ratio operation, and the moving objects are obtained. The emulation experiments demonstrate the validity of the proposed algorithm.