Nonlinear compensation in coherent optical fiber communication represents a crucial challenge for capacity enhancement. Digital backpropagation (DBP) is extensively utilized for nonlinear mitigation owing to its robust theoretical interpretability. Previous research has enhanced the DBP algorithm through various methods, including algorithmic structure refinement and nonlinear operator optimization, each supported by theoretical foundations. However, the performance improvements achieved through nonlinear parameter optimization lack comprehensive theoretical analysis. This paper addresses this research gap by examining the error sources in nonlinear parameter optimization. Based on error analysis, the theoretical model is reformulated to implement nonlinear compensation. The results indicate that the proposed theoretical framework provides substantial validation for the observed performance enhancements.

The investigation begins with an analysis of theoretical errors in the DBP algorithm’s compensation process, examining the ratio between self-phase modulation (SPM) and self-steepening effects before and after sampling through propagation equation derivation and nonlinear operator calculations. The analysis reveals significant inherent errors in this process. The temporal scale variation during sampling causes substantial changes in the SPM-to-self-steepening ratio due to the time-dependent nature of the self-steepening effect. Based on this observation, self-steepening suppression is incorporated into the compensation framework. Through theoretical re-derivation, the nonlinear operator is reformulated, with particle swarm optimization (PSO) applied to enhance compensation accuracy.

The investigation included reproduction of the adaptive DBP algorithm for optimized nonlinear compensation. While the adaptive DBP algorithm represents blind estimation, the extended DBP functions as its interpretable counterpart. Experimental results demonstrate that for quadrature phase shift keying (QPSK) modulation with four-step DBP compensation, the extended nonlinear operator delivers an average gain of 0.96 dB compared to fixed operators, and 1.87 dB improvement over dispersion compensation alone. For 16QAM (64-order quadrature amplitude modulation), the respective enhancements are 0.65 dB and 5.485 dB, while 64QAM exhibits gains of 0.22 dB and 2.06 dB. Notably, the adaptive nonlinear operator achieves similar performance to the extended operator across all three modulation formats. Further tests under varying step sizes and transmission distances consistently demonstrate comparable performance characteristics.

This study examines the theoretical mechanism for parameter optimization of the DBP algorithm in nonlinearity compensation, beginning with the nonlinear Schr?dinger equation. The analysis reveals that performance degradation in parameter optimization stems from the reduction of the nonlinear component in the theoretical derivation. To address this limitation, an extended nonlinear operator is derived from the nonlinear Schr?dinger equation, incorporating both power and amplitude terms. This extended nonlinear operator is implemented alongside the particle swarm algorithm for backend nonlinear compensation and evaluated against DBP and adaptive DBP algorithms. The theoretical mechanism established for the parameter optimization issue in the DBP algorithm aligns with previous theoretical analysis and simulation results.