This study proposes a deep learning-based color hologram reconstruction method to address the issues of complex reconstruction operations, inaccurate color fusion, and zero-order influence during the reconstruction of large objects. The improved U-Net model is used as the network structure, and the spectrum of color off-axis Fresnel holograms generated by mixing actual photography and simulation is used as training samples to achieve the accurate reconstruction of color holograms. Reconstruction experiments are conducted on simulated holograms and actual digital holograms. Moreover, the results have shown that compared to traditional methods, the proposed method can maintain high resolution and color accuracy of the reconstructed image while achieving improved reconstruction results. The outcomes of the study have potential applications in the reconstruction of color holograms in large-scale inspection fields, and are useful for the application of color holographic detection and deep learning in the field of optical imaging.