With the rapid growth of internet traffic, the demand for large transmission capacities from all walks of life has grown dramatically. In the context of limited device bandwidth and high hardware update costs, typical expansion methods include wavelength division multiplexing (WDM) and mode division multiplexing (MDM). At the same time, optical signal impairments caused by devices and fiber optic channels make advanced digital signal processing technology crucial to achieving high-speed fiber optic communications. With the emergence and development of deep learning, equalization based on machine learning has become a hot topic in the field of optical communications. At present, the MDM field mainly uses the intensity modulation direct detection (IMDD) method for experiments and most use a single-channel single-model traditional linear equalizer for channel compensation. However, in scenarios with many mode multiplexing channels and polarization multiplexing, the requirements for nonlinear equalization capabilities are gradually increasing. We adopt WDM, MDM, polarization multiplexing, and advanced digital signal processing technology to construct a homodyne coherent transmission system based on multiple input-multiple output neural network equalizer (MIMO-NNE). We successfully achieve the equalization of 16 channels of 48 Gbaud 16QAM signals after transmitting 100 km of few-mode fiber (FMF) on six modes: LP01, LP02, LP11a, LP11b, LP21a, and LP21b. The bit error rate (BER) of the MDM-WDM system can meet the 15% soft decision forward error correction (SD-FEC) threshold of 1×10^-2.

A 16-channel signal with 50 GHz spacing is generated at the transmitter. The channels are divided into two groups according to odd and even, and each group uses 8 external cavity lasers (ECLs) to couple into optical carriers. After the high-speed signal undergoes the transmitting side digital signal processing(tx-DSP), it is loaded onto the in-phase/quadrature modulator (I/Q MOD) through the arbitrary waveform generator (AWG). A delay line is adopted to divide the optical signal modulated by a single modulator into multiple channels for multiplexing. The odd and even signals are divided into two by a 1×2 optical coupler (OC) respectively and are sent to the polarization multiplexer (PM) after being decorrelated by a delay line, and coupled by a 1×2 optical coupler. After the signal is amplified by an erbium-doped fiber amplifier (EDFA), it is divided into 6 beams by a 1×6 optical coupler, and a delay line is again used to decorrelate LP01, LP02, LP11a, LP11b, LP21a, and LP21b. After being multiplexed by a mode multiplexer, the wavelength division multiplexed signal is transmitted on a 100 km FMF. We use 6-mode EDFA to simultaneously amplify and compensate for each channel mode signal. At the experimental receiving end, after decoupling by the mode demultiplexer, the 6-channel signals pass through a dense wavelength division demultiplexer (DWDM), and an optical switch is applied to gate the 6 wavelengths respectively. Finally, the coherent receiver (CR) performs polarization demultiplexing and homodyne coherent reception. We adopt a real-time digital storage oscilloscope (DSO) to capture the baseband electrical signal and perform offline DSP. In the receiving side digital signal processing (rx-DSP), the precise down conversion (DC) is conducted on signals to compensate for the frequency offset of the system. Then the signals undergo Bessel filtering, resampling, and Gram-Schmidt orthogonalization (GSOP) to solve the problem of IQ imbalance. Additionally, we perform clock recovery (Retiming) to eliminate timing errors and perform chromatic dispersion compensation (CDC). Finally, we adopt the MIMO-NNE to perform channel nonlinear equalization to compensate for nonlinear damage and calculate BER.

Figure 5 shows the BER of the traditional MIMO-LMS algorithm and the proposed MIMO-NNE algorithm under different 6-mode EDFA current. MIMO-NNE algorithm has an average bit error gain of about 0.02 compared to MIMO-LMS algorithm. At the same time, MIMO-NNE algorithm can make the BER of 16QAM transmitted over 100 km lower than the 1.0×10^-2 SD-FEC threshold. Fig. 6 shows the convergence process of the mean square error (MSE) of MIMO-NNE algorithm and MIMO-LMS algorithm under an approximate iteration data amount. MIMO-NNE algorithm has better convergence performance than MIMO-LMS algorithm. As shown in Figs. 7 and 8, the difference in BER of each mode and different wavelength sub-channels is not significant. At the same time, the BER of each channel in 100 km transmission is lower than the 1.0×10^-2 SD-FEC threshold.

In this study, we experimentally build a 6-mode 16-wavelength dual-polarization homodyne coherent transmission system. At the receiving end, the MIMO-NNE based on multi-label technology is used for channel equalization. When transmitting 100 km, the total system rate reaches 36.864 Tbit/s. With the help of MIMO-NNE, the MDM-WDM system BER can meet the 15% SD-FEC threshold of 1×10^-2. The experimental results confirm the nonlinear equalization potential of MIMO-NNE in future high-capacity long-distance transmission systems.