Aiming at the problems of poor hand-time tracking, low recognition accuracy and environmental impact, a hand tracking method based on neural network and Kalman filtering is proposed. The method firstly locates the detection target appearing in the video through the neural network, then estimates the target motion by Kalman filter, compares the estimated result with the detected target in the next frame image, and finally detects the target, tracking and displaying the trajectory of the hand movement in real time. Experiments show that the method can track multiple hand targets in real time and keep tracking when the cross and deformation occur during hand movement. The average processing frame number is 21.212 f/s, and the tracking accuracy rate is 94.88%. It basically meets the requirements of stable, reliable, high real-time and high robustness of hand tracking.