Traditional calibration methods for a Stokes ellipsometer will reduce the calibration accuracy of instrument matrix and effect the measuring accuracy of a polarization state, due to the polarization effect of incident light and the imperfect optical elements in a calibration unit. To measure precisely the instrument matrix of the Stokes ellipsometer, a calibration method was proposed based on the nonlinear least-square fitting method. By taking error sources mentioned above and instrument matrix together as unknown parameters, the function formula between detector intensity and unknown parameters was established based on the Mueller matrix theory. Then nonlinear least-square fitting method was used to fit the detecting intensity curves changed with the azimuth of calibration units and to calculate the instrument matrix of Stokes ellipsometer. The new method and traditional methods were used to calibrate the instrument matrix at 500-700 nm, and it is shown that the total Root Mean Square (RMS) of theStokes parameter by the proposed method is 1.6% in 500-600 nm, about 0.5% less than that of traditional methods. Moreover, when wavelength is larger than 600 nm, the RMS drops to 2.4%, but it still is far less than that of traditional methods. In conclusions, the method is easily feasible and is suitable for the calibration of instrument matrix for various Stokes ellipsometers.