A real-time calibration method is proposed for investigation of tunable optical filter (TOF) nonlinearity. Based on the transmission characteristic of TOF, nonlinearity real-time calibration system is established. Using least mean square algorithm, the relationship between wavelength and voltage of the TOF can be obtained. Thus the nonlinearity of the TOF can be described. The wavelength random error of the TOF is measured by using wavelength blocked F-P etalons. The result shows that wavelength random error of the TOF can be reduced by 37%～95% with reference wavelength. Combined with the method, the wavelength precision in gas pressure sensing system is obviously improved in position accuracy, system error and high sensitivity.