A nonlinear reconstruction method with a single digital hologram using deep learning was proposed for off-axis Fresnel digital hologram. Classic Fresnel diffraction integral is utilized for simulating digital holographic imaging to provide the training samples, and a deep convolution residual neural network is utilized to implement on the object image reconstruction from the recorded hologram, by learning the nonlinear mathematical mapping from the digital hologram to the corresponding object image. The results of numerical simulation experiments show that the method could directly eliminate zero-order images and twin images without fringe pre-processing procedure for extracting object term, compared with the traditional frequency filtering and four-step phase-shift techniques for achieving Fresnel digital holography reconstruction, as well as high quality reconstructed object image. It also has strong robustness to the test dateset generated with different diffraction distances using same recording reference light waveform.