In the 9th laser system of Shenguang II inertial confinement fusion (ICF) facility, tunable Fabry-Perot (F-P) filter is applied to compensate the FM-AM effect. According to the technical requirements of compensation device, a precise displacement monitor system using the capacitive displacement sensor with nanometer scale precision to stabilize the spacing of the tunable F-P filter is proposed firstly, then the basic structure and operating principle of the monitor system are analyzed. The scheme design of the driving circuit of capacitive displacement sensor, the data processing as well as the system controlling program are discussed in detail, and the precision of capacitive displacement sensor is calibrated. The experimental results show that the spacing stability of the tunable F-P filter is better than 15 nm/h and the modulation depth of the FM-AM effect is better than 4% by introducing the displacement monitor system.