In this paper, we propose a new complex-valued dense atrous neural network (CDANN) for phase-only hologram (POH) generation. The network architecture integrates a complex-valued partial convolution (C-PConv) module into the down-sampling stages of dual U-Net structures, enhancing computational efficiency through selective channel-wise processing. To improve feature extraction, we introduce a novel complex-value dense atrous convolution (DAC) module, which employs four cascaded branches with multi-scale atrous convolutions to capture intricate features while maintaining spatial resolution. Additionally, we integrate a spatial pyramid pooling (SPP) module into the U-Net architecture to encode multi-scale contextual features derived from the DAC module. This hierarchical integration expands the U-Net’s receptive field while facilitating cross-layer feature fusion. The proposed method achieves an average signal-to-noise ratio (PSNR) of 32.19 dB and an average structural similarity index (SSIM) of 0.892 within a running time of 24 ms, outperforming conventional approaches. Experiments confirm significant improvements in both reconstruction quality and computational efficiency, making CDANN suitable for real-time holographic displays.