Fourier domain polarization-sensitive optical coherence tomography (FD-PS-OCT) can measure the polarization properties of a sample to provide early diagnosis for some diseases. The accurate diagnosis is required to measure the polarization parameters in a high accuracy. However, the accuracy is greatly affected by wavelength misalignment. Even a slight wavelength mismatch in FD-PS-OCT will result in nonnegligible polarization artifacts. So an accurate wavelength assignment is necessary. In order to decrease the polarization error and improve the accuracy of spectral calibration, the effect of the wavelength misalignment on the polarization parameters calculation should be clear. An analysis is performed on the relationship between the wavenumber misalignment and the calculated error in theory and simulation. Based on the error analysis, a spectral calibration method is proposed. The method achieves the wavelength alignment through evaluating the retardation and fast axis orientation errors. The wave plate experiments show the good performance of the method and the reasonability of the error analysis.