Aiming at the situation of occlusion, light change and complex scenes in current gesture recognition, which easily leads to the decline of the accuracy of gesture recognition, a gesture recognition scheme based on the coupled depth camera of somatosensory control is proposed. In order to improve the accuracy of gesture recognition, the proposed gesture recognition scheme includes Leap Motion data and Kinect depth data. Firstly, the distance between the fingertip and the center of mass of the hand, the height between the fingertip and the palm plane, the angle between the fingertip and the center of the palm, and the 3D position of the fingertip in the hand reference system are extracted from the somatosensory control data. Then, the distance between the finger sample and the center of the hand, the local curvature of the hand contour, the similarity between the distance features and the connected area of the hand shape are extracted from the Kinect depth data. Then, in order to combine the complementary information of two kinds of sensor data and discard redundancy, a joint calibration method is defined by finding the rotation and translation parameters through the 3D position of fingertips collected, and minimizing the average re-projection error of all fingertips in all acquisition frames, determine the external parameters of the somatosensory control sensor and the Kinect depth sensor, and complete the coordinate conversion of the two sensors. Finally, a classification learning method based on multi-class Support Vector Machine (SVM) is proposed. Experiments show that the average recognition rate of Jochen Triesch gesture database is 97%. In different light, skin color and background environments, The proposed algorithm has higher accuracy and robustness than the existing hand gesture recognition algorithms.