The exponential growth in traffic demand within fiber access networks is driven by the rapid advancement of multi-service applications, including 5G/6G, artificial intelligence, and big data. To address this demand, optical communication technologies leveraging multiple physical dimensions—such as space-division multiplexing (SDM), mode-division multiplexing (MDM), and high-order modulation—have been extensively implemented, significantly increasing optical fiber transmission capacity. However, as per-fiber capacity approaches the Pbit/s threshold, complex channel impairments arising from modal coupling, Kerr nonlinearity, and chromatic dispersion become more pronounced. These impairments generate substantial channel imbalance, emerging as a critical performance limitation, particularly in short-reach optical interconnects within data centers.

Intensity modulation with direct detection (IM-DD) continues to serve as a prevalent solution to meet requirements for high integration density and low power consumption. However, restricted modulation bandwidth and in-channel signal degradation pose considerable challenges, necessitating advanced equalization techniques.

Traditional approaches to mitigate channel impairments have followed two primary paths optical-domain equalization utilizing passive fiber components and electrical-domain digital signal processing (DSP). Optical-domain equalization facilitates low-power, high-speed operation through pre-compensation but exhibits high sensitivity to device precision and environmental conditions. DSP-based methods employ algorithmic compensation at the receiver side, including feedforward equalizers (FFE), Volterra nonlinear equalizers (VNLE), and neural network?based equalizers, offering robust and adaptive performance. However, semiconductor processes approaching physical scaling limits render the complexity of nonlinear equalizers increasingly difficult to address through device density improvements. Furthermore, substantial training costs and limited convergence efficiency present significant obstacles to practical implementation. Consequently, developing more efficient and scalable equalization strategies has become crucial.

Advances in silicon photonics have facilitated expanded implementation of photonic chips for optical-domain equalization. These systems reduce DSP complexity and enable low-latency, low-power operation by performing joint linear and nonlinear compensation prior to electrical conversion. Photonic chips incorporating optical neural networks demonstrate capability in learning and compensating for channel impairments, indicating potential for intelligent equalization. However, significant challenges persist, including multi-physics co-design, fabrication tolerance control, and complementary metal oxide semiconductor (CMOS) compatibility. Large-scale integration introduces additional concerns regarding inter-channel crosstalk and thermal instability.

Hybrid optoelectronic equalization, which combines optical speed and parallelism with electronic adaptability, represents an emerging solution for next-generation communication systems. Given the significant nonlinearity in data center transmission channels, a systematic review of hybrid optoelectronic equalization techniques’ working principles and recent developments is essential. A thorough comparison of various approaches’ advantages and limitations provides valuable direction for future research and advances channel equalization toward reduced power consumption, decreased latency, enhanced integration, and improved intelligence.

This paper reviews the operating principles and recent advancements of classical electrical-domain equalization algorithms and neural network?based DSP equalizers, providing a systematic comparison of their respective advantages and limitations. It then shifts focus to current hybrid optoelectronic equalization techniques for optical channel compensation and briefly introduces the main research contributions of our team.

Early optical communication systems utilized fiber components’ physical properties for optical-domain compensation (Fig. 1). However, traditional optical compensation methods prove inadequate for modern high-speed and high-modulation-format networks as data rates exceed 100 Gbit/s. Consequently, DSP-based electrical-domain equalization has emerged as a flexible and programmable solution, capable of reconstructing channel impairments by solving the inverse form of the nonlinear Schr?dinger equation (NLSE). Common implementations include feedforward equalizers (FFE, Fig. 2), decision feedback equalizers (DFE, Fig. 3), and Volterra series equalizers (Fig. 4), achieving linear and nonlinear compensation through adaptive filter coefficient updates.

The increasing distance and capacity of short-reach optical links have resulted in more complex nonlinear channel distortions that resist conventional modeling approaches. Performance requirements necessitate advanced DSP fabrication processes. Traditional equalization algorithms face practical constraints including challenging parameter tuning and limited adaptability to dynamic channels. Neural network?based equalization presents a viable alternative, with researchers exploring standalone neural models and hybrid DSP?neural network architectures for managing linear and nonlinear impairments (Fig. 5). Bidirectional architectures such as BiRNN and BiLSTM demonstrate enhanced performance through superior temporal feature capture of optical channels. Reservoir computing (RC), a simplified variant of recurrent networks, provides additional benefits through sparsely connected reservoirs with fixed internal weights, limiting training to the output layer. This reduction in training complexity makes RC particularly appropriate for resource-constrained intelligent equalization systems.

Optical computing has emerged as an energy-efficient alternative, incorporating signal processing directly within the optical domain and circumventing traditional electronic DSP architectures’ power and bandwidth constraints. Recent advances in silicon photonics enable optical neural networks (ONNs) to integrate deep learning capabilities with photonic circuits, facilitating intelligent equalization and pre-compensation entirely within the optical domain. The extension of reservoir computing to the photonic domain creates photonic reservoir computing (Fig. 6), combining minimal complexity with robust nonlinear dynamic processing capabilities.