In this paper, machine learning method is applied to 30 Tbit/s (60 × 500 Gbit/s) Nyquist Dual Polarization-16 Quadrature Amplitude Modulation (DP-16QAM) system after 6 300 km transmission in G. 654E optical fiber. Nonlinear channel equalization is used to reduce the transmission Bit Error Rate (BER).

Referring to the "receptive field" mechanism of convolution neural network, the size of "convolution core" is designed, and the data set is constructed according to the divided sampling data. The artificial neural network is constructed by optimizing the parameters. The one-to-one data corresponding to the transmission and reception of different wavelengths, different optical signal-to-noise ratios, and different fiber input powers in the C-band are collected. Refer to the classic full-connection neural network structure, the neural network is constructed according to the data structure of the data set. The network fitting is carried out for the real part and the imaginary part respectively. After training stage, the test data is sent into the network, and the performances are compared with the traditional methods.

Two kinds of neural networks are used to fit the transmission BER under 60 different wavelength transmission conditions of C band frequency from 191.562 5 to 195.987 5 THz. Compared with Maximum Likelihood Sequence Estimation (MLSE), Network 1 has an average reduction of 23% in BER, and Network 2 has an average reduction of 41% in BER. A frequency of 193.812 5 THz is then selected for the calculation of the fiber input power ranging from 14 to 19 dBm. The average improvement in network 1 and network 2 are 32% and 52%, respectively. Under different optical signal-to-noise ratios, Network 1 has an average improvement of 30%, and Network 2 has an average improvement of 57%.

The two neural networks have excellent performance in nonlinear equalization of coherent transmission systems. At the same time, the number of network layers and nodes will jointly affect the fitting results. Increasing the number of layers and nodes can obtain better fitting results, but the corresponding parameters, training time and the required space will also increase. Therefore, in the application, the actual situation should be considered to choose between the fitting performances and the model attributes.