Digital holography (DH) is critical for monitoring quantitative three-dimensional information of transparent samples. However, phase aberration compensation and unwrapping are needed in conventional digital holographic reconstruction, which adversely affect its speed and accuracy. We propose an improved residual Unet method that combines dilated convolution and attention mechanism to implement end-to-end phase reconstruction of DH, which simplifies the imaging process and improves the quality of image reconstruction. In addition, the proposed method can further optimize the network model for real-time reconstruction by adjusting residual blocks. The experimental results reveal that the proposed phase reconstruction method based on deep learning can obtain accurate three-dimensional information of samples in real time, which benefits real-time monitoring for dynamic samples.