Based on simulation of the impact of gamma nonlinearity of the projector on phase error, an algorithm is proposed to build a look-up table (LUT) for phase error compensation by analyzing the projector grating directly. In this algorithm, a set of grating images of a uniform flat surface board are captured and the relation of grating phase and phase error in established. Then the relation is analyzed and stored in a LUT for phase error compensation. Experimental results show this algorithm reduces greatly the phase error caused by the gamma nonlinearity of the projector. The measurement precision of the system reaches 0.043 mm, 5.6 times higher than that before error compensation.