Back propagation (BP) algorithm is a common effective method for nonlinear compensation in high-speed fiber optic system. System performance can be improved a lot while the computational complexity can be reduced in the same time if a Gaussian filter is added to BP. However, the Gaussian filter is an ideal filter and it′s hard to be designed with its complex structure. The scheme that the window finite impulse response (FIR) filter is combined with BP algorithm is proposed. The compensation performance influenced by the filter parameters selection is analyzed in detail, and the performances of different schemes in the 640 km non-dispersion compensation fiber transmission system with 224 Gb/s dual-polarization 16 quadrature amplitude modulation (DP-16QAM) signal are compared. The results prove that the frequency domain characteristics of Hanning FIR filter are similar to the Gaussian function, which can compensate the nonlinear damage and improve the system performance effectively with only 9 taps. Results of the experimental system show that the proposed FIR-BP algorithm can reduce the bit error rate (BER) to 10-3 or less and can increase the optimum fiber input power at least 3 dB compared to the linear compensation algorithms.