Moving cast shadow usually is misclassified as foreground. In order to solve the problem, a novel method is proposed to detect moving cast shadow based on energy minimization. Firstly, the stationary background and moving objects could be obtained by the traditional pixel-wise method. Then, the color and texture invariance of cast shadow is employed with the temporal-spatial coherence to construct the energy function through Gibbs energy. Finally, moving cast shadow and foreground are segmented accurately through minimizing the energy function. The method is tested with some videos including moving shadow and the results show that the proposed method excels previous methods both in indoor and outdoor scenes.