Fig. 1. LSTM neural network equalizer. (a) Schematic diagram of LSTM neural network equalizer; (b) structure diagram of LSTM cycle unit

Fig. 2. Transmission system block diagram and signal processing flow. (a) Diagram of 4 mode 80 channel dual-polarization homodyne coherent transmission experimental device; (b) transmitting DSP flow; (c) receiving DSP flow

Fig. 3. diagram of transmission experiment

Fig. 4. Comparison of BER obtained by various equalizers at different transmission distances

Fig. 5. MSE reduction process of MIMO-LMS, NNE, and LSTM-NNE training

Fig. 6. Measured BER under different mode subchannels with illustration showing constellation diagram corresponding to X polarization in LP21a mode

Fig. 7. Measured BER under different wavelength subchannels in 650 km transmission