The laser triangulation method has been widely applied due to its advantages in non-contact measurement, large measurement range, high accuracy in measurement, simple structure, high adaptability, etc. However, the theoretical formula of laser triangulation is nonlinear and the parameters(a, b, θ) of optic structure in practical engineering is immeasurable. Therefore, we investigated the construction approach of the mathematical model in triangulation, and adopted the polynomial expansion approach to build the model. Under the application of least square fitting polynomial method of the solution to the model coefficient, we proposed a evaluation method according to the requirements of the maximum relative fitting residual combined with the relative coefficients for controlling the fitting polynomial order. The feasibility of the method was verified by the practical optical system, with the relative error of 0.01%. The method of least square fitting polynomial has practical guiding significance for the calibration of laser triangular displacement sensor and the elimination of systematic error.