The sun is a huge plasma laboratory, in which all phenomena and processes are caused by electromagnetic interaction. Therefore, the observation and research of the solar magnetic field play an important role in solar physics and space weather science. At present, the measurement of solar magnetic field is mainly based on the Zeeman effect and its polarization variation in a narrowband spectrum. Therefore, for imaging equipment, the solar magnetic field measurement requires a filter with a bandwidth of about 0.01 nm. High precision and high sensitivity tunable narrowband imaging is one of the core technologies for solar telescopes to achieve accurate magnetic field observation. Lyot birefringent filter is an imaging filter invented by Lyot Bernard in 1933. It realizes ultra-narrow band filtering through the interference effect of high birefringent crystals, and is the most commonly used narrow band imaging device for solar telescopes. Traditional birefringent filters use rotating wave plates to adjust the central wavelength of transmission. However, mechanical modulation is not only slow, but also faces reliability risks such as life, bubbles, and oil leakage when applied to space telescopes. Based on this, with the rapid development of liquid crystal modulation technology in recent years, tunable liquid crystal birefringent filters are gradually favored. Instead of mechanical modulation, Liquid Crystal Variable Retarder (LCVR) is used to achieve wavelength adjustment through transmission, which is fast and free of risks such as bubbles and oil leakage, significantly improving the scientific performance and reliability of the filter. However, since LCVR is a liquid crystal electro-optic modulator, when tunable liquid crystal birefringent filters are applied to space telescopes, the retardation-voltage curve of LCVR may drift due to the combined effects of radiation, force, heat, charging and other factors, which may lead to divergence of the transmission profile of the filter and the drift of the central wavelength, thus reducing the measurement accuracy of the filter. Therefore, how to realize the in-situ calibration of tunable liquid crystal birefringent filter is the core problem. In 2014, Mudge Jason and Tarbell Theodore proposed a Fourier in situ calibration method, but this method only gives a theoretical formula of the first stage filter, and assumes that all stages of filter are completely linear independent, and its measurement error increases significantly with the retardation shift of LCVR. Therefore, this paper proposes a nonlinear fitting calibration method based on Fourier analysis. Based on the theory of polarization interference, Fourier analysis is used to obtain the phase offset of each stage of the birefringent filter; then, a functional model of the phase offset and the retardation drift of LCVR is established; finally, the nonlinear fitting method is used to obtain the retardation drift, so as to achieve the in-situ calibration of the birefringent filter. The simulation results show that the measurement accuracy of the new calibration method is within 0.1° when the retardation drift of LCVR is within ±45° and the central wavelength drift of the prefilter is within the range of [-0.17 nm 0.07 nm]; when the thickness manufacturing error of calcite crystal is within ± 10 μm, the retardation measurement error is within 1°; when the first order dispersion change of calcite crystal is within ± 10%, the measurement error is within 2°. In order to verify the validity of the calibration method, a tunable liquid crystal birefringent filter is experimented in Huairou Solar Observation Station (HSOS) in National Astronomical Observatory of the Chinese Academy of Sciences. Through experimental research, both methods are compared. The results show that the LCVR drift is within -40°, the measurement error of the nonlinear fitting method based on Fourier analysis is within 5°, and the corresponding birefringent filter (bandwidth 0.01 nm) center drift is less than 0.000 3 nm; The measurement error of Fourier method is within 20°. When LCVR offset reaches -60°, both calibration methods are invalid. The experimental results are basically consistent with the simulation analysis.