This paper proposes an improvement method for addressing the low reconstructed-image quality and insufficient computational efficiency in three-dimensional hologram calculation for the random phase-mask-extension holographic-display viewing-angle method. The improvement includes two aspects: 1) Introducing the calculation of band-limited angular spectrum diffraction into the Gerchberg-Saxton (GS) phase-iteration process to enhance the reconstructed-phase hologram quality after expanding the viewing angle using an additional random phase mask and effectively suppressing speckle noise in hologram-reconstruction results using the time-averaging method; 2) optimizing the GS phase-iteration process by combining a lookup table design with a multi-compute unified device architecture (CUDA) stream parallel algorithm to significantly improve hologram-generation efficiency. Numerical-simulation results show that the introduction of band-limited angular spectrum diffraction into the GS algorithm combined with the time-averaging method yields higher imaging quality compared with the conventional GS algorithm combined with time-averaging method. Compared with the results of the conventional GS optimization, the proposed method reduces the speckle contrast from 0.41 to 0.2065, increases the peak signal-to-noise ratio (PSNR) from 10.6808 to 18.0546, and improves the structural similarity (SSIM) from 0.2778 to 0.7338 in the reconstructed results. Meanwhile, using the hierarchical hologram acceleration scheme designed in this study, the computation time for a single three-dimensional object hologram is reduced from 1410.53 s to 16.47 s, thus demonstrating the feasibility of the proposed method.