Coherent optical communication is a main research focus in fiber optic communication and is suitable for modern communication networks due to its large bandwidth and long transmission distance. For long-distance fiber optic communication systems, increasing the launch power can improve the transmission distance and system capacity, but it will also result in the enhancement of nonlinear effects. With the rising transmission distance and communication rate of today’s communication systems, increasing the launch power not only improves the system capacity but also causes more obvious nonlinear effects. The digital back propagation (DBP) algorithm can compensate for the noise in optical fibers, but its high complexity limits its applications in coherent optical communication. Therefore, we propose a design method for the step size of the DBP algorithm for power equalization and introduce an independent power compensation factor for each step. The results show that both the step length design method and the introduced power compensation factor improve the ability of the DBP algorithm to compensate for nonlinear effects.

We focus on the nonlinear effects in optical fibers and investigate methods to compensate for them. Firstly, we investigate the DBP algorithm, which utilizes DSP technology to programmatically construct a virtual link in the digital domain with the same length as the real transmission fiber link but with opposite transmission parameters (loss, dispersion, and nonlinear coefficients), thus realizing the damage compensation in the optical fiber. Then the iterative process of the DBP algorithm is studied, the existing step-size design algorithms are derived, and numerical simulations are carried out in accordance with the formulas to analyze the power fitting of different step-size design algorithms in carrying out the iterations, with a step-size design method considering the power fitting being proposed. For the proposed power equalization step, an independent power compensation factor is also introduced for each step and optimized by adopting a genetic algorithm. Finally, the algorithms using different step sizes are simulated to verify the compensation performance difference of various algorithms.

The proposed equal power step DBP (EP-DBP) has better nonlinear compensation performance than the existing DBP algorithms. Under the same transmission power, the Q factor of logarithmic step DBP (LS-DBP) and equal power step EP-DBP are higher than that of constant step DBP (CS-DBP), and the optimal transmission power is improved, which indicates that the optimized step size is effective in improving the compensation performance. Under different modulation formats, the Q factor of EP-DBP is improved by 0.37 dB, 0.50 dB, and 0.57 dB over CS-DBP, and that of EP-DBP is improved by 0.21 dB, 0.18 dB, and 0.29 dB over LS-DBP, respectively.

We propose a nonlinear coefficient optimization method based on equal power DBP, which includes a step design method based on equal power and a power compensation coefficient optimization method based on genetic algorithms. Meanwhile, a remote fiber optic communication system is established, and various DSP algorithms are employed to compensate for the receiver end respectively, with the performance of different algorithms compared. The results show that DBP outperforms EDC in nonlinear compensation, and EP-DBP outperforms CS-DBP and LS-DBP in nonlinear compensation, with Q factor improvements ranging from 0.37 dB to 0.53 dB and 0.18 dB to 0.29 dB respectively. Additionally, EP-DBP also reduces the complexity by about 50% without performance loss, and under larger step sizes, its compensation is more favorable. The improved power compensation factor optimization method based on power iso-distribution can effectively lower the complexity of the DBP algorithm and improve the nonlinear compensation effect.