Light field technology can boost image encryption technology from two-dimensional to three-dimensional, and enhance the security of encryption. The refocusing algorithm can be used to achieving image decryption. However, it will introduce interference between images. Based on the deep learning technology, the regularity of image interference is analyzed. The simulated light field data set is constructed. This paper creats a 7-layer full convolutional neural network. As for training the full convolutional neural network, the simulated light field data set is used as input, while the original images are used as labels and input into the full convolutional neural network. Then the real light field decrypted images are input to for testing. The experimental results show that the full convolutional neural network can decrease the interference of the optical field decrypted images obviously and improve the image quality effectively.