In order to improve the measurement accuracy of FBG sensing system, particle swarm optimization (PSO) algorithm combined with reference FBGs array was applied to investigate the nonlinearity and hysteresis character of Fabry-Parot filter (FPF). A method of modeling the wavelength-voltage relationship of FPF online in each FPF scanning cycle was proposed in the present paper. The feature of particle swarm optimization algorithm such as fast convergence and simple implementation makes the process of modeling wavelength-voltage relationship of FPF be completed with low computing cost and high accuracy. With the set-up model, the absolute error in wavelength detection of FBG sensors was demonstrated by experiments to be as low as 0.03 nm. The structure of the system is compact and the proposed modeling approach has important meaning in FBG sensors system when FPF is used as wavelength demodulator.