In view of the problem of pigment selection in the restoration of cultural relics such as murals and painted sculptures, terahertz time-domain spectroscopy is used to distinguish the common heritage pigments. First, the time-domain spectra, frequency-domain spectra, transmission spectra, absorption spectra, and refractive index of six pigments with very different colors (garcinia, loess, laterite, lapis lazuli, calcite, and mineral green) are analyzed and studied in the range of 0.2?1.5 THz. Then, to further demonstrate the applicability of terahertz time-domain spectroscopy, the terahertz spectra of garcinia and orpiment and those of laterite and sinopis, which have very similar colors, are analyzed. Finally, the experimental results show that the spectra of the pigments have evident differences in the spectral characteristics of pigments with the same main ingredients, which are manifested by the shift, increase, and decrease of absorption peaks. Based on these characteristics, the pigments with the same main composition can be accurately identified. The results indicate that the proposed terahertz measurement techniques can provide accurate reference for the restoration of color of cultural relics.