To optimize the signal demodulation quality, spatial resolution, and vibration localization performance of the phase-sensitive optical time domain reflectometry (Φ-OTDR) system, this paper investigates the effect of the low-pass filter (LPF) cut-off frequency on system performance based on the heterodyne coherent detection of the Φ-OTDR system with under-sampling and in-phase and quadrature(IQ) demodulation techniques. In the experiment, the narrow linewidth laser, the acoustic optical modulator, the balanced photoelectric detector and other devices are used to construct the system, and five LPFs with different cut-off frequencies(12.5, 10, 7.5, 5, 2.5 MHz) are set for comparative analysis. The results show that when the LPF cut-off frequency is 7.5 MHz, the signal-to-noise ratio (SNR) of the demodulated signal reaches the highest value (40 dB), and the signal quality is more stable between different sampling points. Meanwhile, the LPF cut-off frequency has no obvious effect on the spatial resolution of the system, but there is an error on the vibration localization(about 2~5 m).