Phase-shifting technique is commonly applied in surface measurement of three-dimensional object in recent years. In phase measuring profilometry detector nonlinearity is the main factor introducing the errors of the measurement . This paper gives the theoretical analysis and the computer simulation of this kind of error,and compares the ability of tolerance with the second and the third order nonlinearity of detector for N-frame phase shifting algorithms. Also,this paper gives several geometry diagram to show intensity of ray when there is the nonlinearty error. The results of numerical simulation and theoretical analysis are consistent and it offers a theoretical basis of selection of algorithms.