The rapid advancement of technologies such as cloud computing, the internet of things (IoT), and artificial intelligence has driven exponential growth in global data traffic. By 2025, mobile phone users will constitute approximately 70.5% of the global population, with internet users reaching 5.56 billion, representing a 2.5% increase from 2024. Data traffic is projected to increase 2.5 times from 2024 to 2030. To address these demands, optical communication technologies continuously advance transmission capacity through innovations such as multi-dimensional multiplexing and advanced modulation formats. However, nonlinear effects in optical fibers, including self-phase modulation (SPM), cross-phase modulation (XPM), and four-wave mixing (FWM), combined with linear impairments like dispersion and polarization mode dispersion (PMD), present significant challenges to long-distance, high-capacity transmission. These effects distort signal waveforms, increase bit error rates, and fundamentally constrain transmission capacity and distance, creating critical challenges for next-generation ultra-high-speed systems. Traditional physical-layer compensation methods demonstrate limitations in dynamic adaptability and computational efficiency, necessitating the development of intelligent nonlinear equalization (NLE) techniques. Addressing these challenges remains essential for advancing next-generation ultra-high-speed optical communication systems, as they directly influence the reliability and scalability of global data infrastructure.

Traditional methods for nonlinear compensation in fiber optic systems include digital backpropagation (DBP), perturbation theory-based nonlinear compensation (PNC), and Volterra series methods. DBP, proposed in 2008, compensates for transmission impairments by creating a virtual link in the digital signal processing (DSP) at the receiver end. Despite its high computational complexity, DBP has been optimized through approaches such as weighted DBP (WDBP), improved DBP (iDBP), and subband-processed enhanced split-step Fourier method (CB-ESSFM). These optimizations have significantly reduced computational overhead while maintaining or even improving performance, making DBP a benchmark for nonlinear compensation. PNC leverages perturbation theory to approximate nonlinear effects, offering low complexity and high performance. Recent advancements include triplet-correlative PNC (TC-PNC), which reduces computational complexity by sharing intermediate results in triplet calculations, and second-order PNC (SO-PNC), which extends perturbation theory to higher orders for better compensation accuracy. These methods demonstrate effectiveness in balancing computational efficiency with nonlinear distortion mitigation.Volterra series methods model nonlinearities using multi-order kernel functions, providing a flexible framework for analyzing and compensating nonlinear distortions. Improvements such as symmetric Volterra series nonlinear equalizers (symVSNE) and cascade structures have reduced complexity while maintaining performance, making Volterra methods suitable for high-order modulation formats and long-distance transmission systems.Artificial intelligence (AI) has emerged as a transformative approach, with deep neural networks (DNN) and hybrid architectures demonstrating significant potential. Learning-based digital backpropagation (LDBP) optimizes DBP steps using neural networks, enabling joint optimization of linear and nonlinear compensation parameters. Neural networks combined with perturbation methods (NN+PNC) enhance compensation performance by addressing traditional perturbation approximation limitations. Advanced architectures like long short-term memory (LSTM), convolutional neural networks (CNN), and Transformer-based models address temporal and spatial dependencies in nonlinear distortions. Notable advancements include physics-informed neural operators (PINO), which integrate physical laws with data-driven learning to ensure consistency with fiber optic transmission principles, and meta-learning approaches that enable rapid adaptation to new channel conditions with minimal training data. These innovations advance nonlinear equalization, offering robust solutions for next-generation optical communication systems.

The research of nonlinear equalization techniques has evolved significantly, transitioning from traditional methods to data-driven and physics-informed machine learning approaches. While existing methods have enhanced compensation performance and reduced complexity, challenges persist in balancing computational efficiency, dynamic adaptability, and generalization across diverse channel conditions. Furthermore, numerous new nonlinear compensation methods remain in the research phase, necessitating the development of practical nonlinear compensation methods for next-generation optical transmission systems.Future research should prioritize several key directions to advance NLE toward practical implementation and broader applicability. First, the development of efficient architectures remains essential to address the computational limitations of current algorithms. Second, physics-embedded learning paradigms warrant emphasis to bridge the gap between data-driven and model-driven approaches. Specifically, incorporating the nonlinear Schr?dinger equation (NLSE) or Manakov equations directly into neural operators—through physics-informed neural networks (PINN) or differentiable solvers—will enhance model interpretability, reduce training data requirements, and improve generalization across diverse fiber parameters. Third, achieving cross-scenario robustness requires innovative adaptation mechanisms, particularly for dynamically reconfigurable optical networks. Transfer learning techniques could adapt pre-trained NLE models for novel fiber types (e.g., hollow-core or multi-core fibers) with minimal retraining, while federated learning architectures may enable collaborative model optimization across geographically distributed network nodes while maintaining data privacy. These advancements will accelerate the transition from laboratory prototypes to field-deployable solutions, facilitating the development of terabit-per-second fiber-optic networks.