Aiming at the problems of polarizing effect in short wave infrared and measurement deviation in the polarization spectral measurement system built in lab, a system calibration model was established and a method to measure the correction coefficients of the model was proposed, thus the polarizing effect was calibrated precisely. Firstly, the main problem of the regular time-divided polarization spectral measurement system was described and the origin of the polarizing effect was analyzed. An ASD spectrometer which had a serious polarizing effect functioned as an analyzer according to the polarization transmission theory. Thereby a system calibration model was established and verified by comparing with the ideal model widely used. Then, a high precision fitting method of the correction coefficients was proposed for improving the measurement accuracy based on the analytical method. Experimental results indicate that the calibration precision is higher than 0.5%, which can fully satisfy the requirements of target polarization information processing in laboratory. This method has a great significance for the development and calibration of polarization spectra measurement system.