The traditional Gaussian Mixture Model (GMM) is built based on every single pixel in RGB color space, which leads to inaccurate detection results, trailing smear and inanition inside the moving objects. The improved GMM is built in HSV color space, which is fit for human visual system. Single pixel is taken place by a vector which is composed of central pixel itself and its neighbor pixels in order to improve the performance of the model. The connected area is extracted through color segmentation method in order to make full use of the color information. Finally, the shadow area is restrained by Phong’s object lighting model. According to the results of experiments, the improved algorithm can detect moving objects much more precisely. Compared with the traditional GMM, it is robust to lighting changes and shadow.