A new fire detection algorithm is proposed for solving the problems of the existing video image fire detection algorithm, such as serious loss of foreground information, high false alarm rate and weak generalization ability. It mainly consist of two parts including foreground extraction and classification decision. In order to extract more accurate foreground region, an improved ViBe algorithm is applied to obtain the selectively updated motion area. Meanwhile the color features in the motion area are classified with a two-stage classifier composed of random forest and support vector machine. In the classification decision module, two novel kinds of early flame features are suggested to describe the ratio of the inter-frame area overlap rate to the intensity of different sections movement in the flame region, and then combined with the Hu moment feature for training the decision classifier. The experimental results show that the algorithm is more adaptable for practical applications with high accuracy, low false alarm rate, strong generalization ability and short response time.