Fig. 1. Schematic diagram of MIMO-NNE

Fig. 2. Transmission system block diagram and signal processing flow. (a) 6-mode 16-channel dual polarization homodyne coherent transmission experimental device; (b) transmitter side DSP（tx-DSP）; (c) receiver side DSP（rx-DSP）

Fig. 3. Schematic of 6-mode EDFA

Fig. 4. Average gain of 6-mode EDFA under different incident optical powers

Fig. 5. Performance comparison of MIMO-LMS algorithm and MIMO-NNE algorithm under different 6-mode EDFA currents

Fig. 6. Mean square error reduction process during MIMO-NNE algorithm and MIMO-LMS algorithm training. (a) MIMO-NNE algorithm; (b) MIMO-LMS algorithm

Fig. 7. Measured bit error rate under different mode subchannels

Fig. 8. Measured bit error rate under different wavelength subchannels