In long-distance optical communication systems, compensating fiber nonlinear impairment through traditional Digital Signal Processing (DSP) is difficult due to intractable interactions between Kerr nonlinearity, chromatic dispersion and amplified spontaneous emission noise. Machine learning algorithm can be used to further process signals on the basis of traditional DSP to mitigate fiber nonlinear impairment and improve long-distance transmission performance.

In this paper, the traditional Digital Back Propagation (DBP) algorithm is combined with Deep Neural Network (DNN), where the linear step size and nonlinear step size in DBP are taken as one neuron. It means that the linear step size is taken as the weight matrix of DNN, the nonlinear step size is taken as the activation function, and DSP is taken as the static layer of DNN. A DNN-based Learned Digital Back Propagation (LDBP) algorithm is proposed.

In order to verify the feasibility of the proposed LDBP algorithm, the simulation was carried out in a single-channel polarization division multiplexing 16-ary Quadrature Amplitude Modulation (QAM) optical transmission system. The numerical simulation results demonstrate that the 1-step-per-span LDBP algorithm improves the optimal launched power from -2 dBm to 1 dBm in compared to linear equalization. Meanwhile, compared with DBP with the same computational complexity, the proposed algorithm improves Signal-to-Noise Ratio (SNR) by 0.82 dB at the transmission distance of 1 200 km. In addition, compared with DBP with the same computational complexity and linear equalization, the SNR of the transmission system corresponding to LDBP method decreases more slowly with the increase of transmission distance, and the algorithm can work without knowing the link parameters, showing the characteristic of universality and robustness.

The proposed LDBP algorithm is more suitable for practical long-distance coherent optical communication system than the traditional DBP algorithm.