Chaotic encryption is widely used in the field of image encryption due to its initial value sensitivity， pseudo-randomness， and unpredictability of motion trajectories. Deep learning (DL) as a method of machine learning was proposed for decades. With the development of computer’s performance， the practicality of deep learning has been proved more and more. It has achieved good resultsin many fields. In this paper， we propose to attack Lorenz chaotic encrypted system of grayscale images by the deep learning method via Residual Networks. After the training process to a series of input and output plaintext-ciphertext pairs， ResNet can fit the process from ciphertext to plaintext. We can finally recover the image approximate to the plaintext image accordingto the ciphertext which is independent to the original plaintext-ciphertext pairs set. Numerical simulation has verified that the result recovering from the chaotic encrypted system is very good.