In order to avoid the problems of complicated steps and noise interference of traditional holographic reconstruction methods, an improved semantic segmentation U-Net network was used for super-resolution reconstruction of holograms. Firstly, a novel end-to-end neural network was introduced to fully acquire more semantic information of images and to enhance the performance of network learning. Secondly, the efficient channel attention (ECA) of deep neural convolutional network was added to improve the ability of focusing on details in the holograms, and to further improve the accuracy of the network. The leaky rectified linear units (LeakyReLU) was used as the activation function to accelerate the network convergence. Using the low resolution holograms of blood cells and chicken blood cells for training, the super-resolution reconstruction intensity and phase map were obtained. The results show that the improved network can quickly reconstruct the phase and intensity images with rich details, clear edge texture and flat background. The structure similarity index measure (SSIM) and peak signal-to-noise ratio (PSNR) of the blood cell intensity reconstruction images are 0.9613 and 27.38, respectively. Meanwhile, the holograms of different scales can be reconstructed. This study provides a reference for using deep learning to improve the quality of holograms.