Aiming at the influence of low-pass filter in optical fiber nonlinear compensation on the algorithm complexity and compensation accuracy of digital back propagation(DBP), the nonlinear compensation characteristics of 256 Gb/s polarization multiplexing 16 order quadrature amplitude modulation(PDM-16QAM) signal with Bessel, Gaussian and Butterworth low-pass filters are studied. The experimental results show that when the step size is small, the influence of three low-pass filters on DBP algorithm is basically the same, when the step size is large, the performance of high-order Bessel low-pass filter is the best. In addition, when the order of the low-pass filter is 1 and the 3 dB bandwidth is fixed at 8 GHz and the step size is greater than 40 km, the Q factor of the Gaussian low-pass filter is about 0.1 dB better than that of the other two low-pass filters.