In order to obtain color fidelity of reconstructed image before the computer generated rainbow hologram printed and reconstructed optically, a color difference test method based on numerical reconstruction is proposed. Firstly, the computer generated rainbow hologram is analyzed in special frequency domain. The relationships between reconstruction parameters and special frequency distribution are concluded. Then, the spectrum distributions of reconstructed images are obtained through simulated reconstruction of computer generated rainbow hologram by filtering in frequency domain. Moreover, the color difference of reconstructed image is calculated in CIE1976UCS uniform color space. Seven color cards are designed and with its computer generated rainbow holograms. The color differences are tested with the proposed method.The optical experiments are designed with the illumination of halogen light source. The results and analysis are presented and show that the proposed method is effective and fast way for the analysis of spectral distribution and color difference of computer generated rainbow hologram.