Due to the complexity of training model and huge training set of deep learning, the development of deep learning is seriously hindered. We use an open-source platform called TensorFlow developed by Google to build deep learning model for video object recognition and tracking. Some basic theories are introduced including the principles of deep learning and TensorFlow′s properties. The framework of deep learning model developed by TensorFlow is outlined. Experiments are designed based on the standard data in VOT2015. Experimental results show that the model has high computational efficiency and recognition accuracy, and it can adjust network structure easily, find optimal structural model fast and complete video object recognition and tracking task well.