In order to realize real-time gesture recognition based on Kinect and to reduce the recognition time while ensuring the recognition accuracy, we propose a method of gesture image extraction based on Kalman filter, and study a gesture recognition model based on three characteristics. We get depth images and skeleton information via Kinect, and then extract hand regions based on Kalman filter. In order to verify the efficiency of gesture segmentation, we collect 28000 samples of 10 types of gestures, extract two local binary pattern features and histogram of oriented gradient (HOG) feature, and classify the samples by support vector machine (SVM). The experimental results show that the gesture recognition model based on HOG+SVM has the recognition accuracy of 97.09% and the recognition rate of 31 frame/s. In application based on Kinect, HOG+SVM recognition model based on the proposed segmentation method can meet the real-time requirement.