To reduce the phase error caused by the nonlinear response in the digital phase-shifting fringe projection system, a self-correction method based on the universal rational polynomial model is proposed. The method first uses polynomials to model the unknown nonlinearity of the projector, and then establishes the phase error function. Second, the outlier detection algorithm is used to eliminate the deviation points, and the phase error coefficient in the error function is solved iteratively by using the optimization function. Finally, the phase with error is substituted into the phase error function, and the phase error is corrected point by point through fixed point iteration. The experimental results show that the proposed self-correction method is easy to implement and does not require pre-calibration, and can effectively reduce the nonlinear errors in the digital phase-shifting fringe projection process without the need of data calibration in advance, which is helpful to improve the measurement accuracy and efficiency of the system.