A three-dimensional (3D) tracking registration method is proposed based on the semantic object matching. The improved single-shot multi-box detector (SSD) deep convolution neural network is used to segment images semantically and thus the pixel level semantic segmentation results for different objects in the scene are obtained. To solve the object function of the camera pose, the camera pose is estimated by the combination of the gray and the geometric constraints of images. The proposed method not only reduces the influence of the lack or mismatch of feature points on the performance of 3D tracking registration algorithm, but also it can adapt to the scenes with different structures. The research results show that the error of this proposed method is less than 2.2 pixel, which basically satisfies the requirement of real-time.