For the problem of spectral distortion in all fiber Fourier transform spectrometer caused by the phenomenon of polarization fading on the output interferometric signal,the effect of optical fiber birefringence which were induced by stress,bending and twisting to light polarization state were analysed by Polariztion Optics theory,and find the reason of spectral distortion in FFTS.Finally,in order to stabilize the fringe visibility,the method to establish feedback control system of polarization state was proposed.Experimental results demonstrate that the impact of polarization state change has been eliminated by the method.