A generative adversarial network combining color prediction and semantic perception is proposed to address the problem of color haloing and region mis-coloring in texture details after colorization due to large modal differences between NIR and visible images in the colorization process. A dual branch generator is designed for color prediction and semantic perception, where the color prediction branch uses a residual network with jump connections and the semantic perception branch uses a dilated convolutional pyramid structure with semantic fusion. Different expansion rates can obtain multiple perceptual fields to extract multi-scale semantic features, and embed the perceived semantics into the color prediction branch to improve the semantic understanding of the model and improve the color haloing and region mis-coloring problems. A cyclic consistent semantic loss function is designed to constrain the consistency of semantic information in the generator. The algorithm performs performance experimental comparison as well as ablation experiments on an RGB-NIR scene dataset. Experiments show that this algorithm outperforms existing colorization algorithms in terms of PSNR, SSIM, and LPIPS evaluation metrics, and the coloring effect is more consistent with visual perception.