Based on monotonous growth characteristics of the phase of grating images, a monotonous smoothing algorithm based on least square method is proposed for phase error compensation. The monotonous characteristics and error of phase are analyzed in detail. Then a monotonous smoothing algorithm based on developed least square method is used to correct phase error of absolute phase diagram. Experimental results show that the gamma nonlinearity of the projector is reduced by more than 60%, and the corrected absolute phase has the advantage of good smooth.