Screen communication is a near-field communication mode using dynamic bar code as the information carrier and visible light as the media. Due to its strong anti-interference ability, spectrum resources free, and simple deployment, it has a wide range of application scenarios and great development potential. We have proposed a screen communication location tracking algorithm based on the method of deep learning. The information area is processed by Faster Region with Convolutional Neural Networks Feature (Faster R-CNN) algorithm which makes the optical camera can intelligently locate the carrier area without relying on the traditional positioning finder pattern, resulting in the improvement of the single frame carrying. The Lucas-Kanade (LK) optical flow is used for position change estimation of the subsequent frame information region introduced by the jitter, which improves the continuous frame processing rate and the system communication rate. Experimental results show that the mean Average Precision (mAP) of the Faster R-CNN is reached up to 90.91% while the LK optical flow was introduced to improve the processing efficiency of the system, and overcome the bottleneck of the processing power of the terminal system. The processing time was shortened by 59.5% compared with the Faster R-CNN algorithm.