Digital coherent transceivers and complex modulation formats have further advances in optical fiber communications. High-order quadrature amplitude modulation (QAM) coherent optical communications have potential with high spectral efficiency for large-capacity transmission and have been demonstrated in lab experiments and field trials^[1–3]. Meanwhile, probabilistic shaping has been utilized by reason of adaptable rate flexibility and superior energy efficiency^[4–7]. However, the conventional well-developed digital signal processing (DSP) algorithms for uniform low-order QAM signals face some challenges in the probabilistically shaped (PS) high-order QAM signals, and it is indispensable to optimize the conventional DSP algorithm for achieving desired performance.

Updated DSP algorithms for PS QAM signals have been investigated in clock recovery, frequency offset compensation, carrier phase recovery, and channel equalization^[8–11]. In order to improve the sensitivity of a conventional Gardner timing error detector (G-TED), the modified G-TED algorithm balances the effect of PS magnitude and pulse roll-off factor^[8]. Two blind frequency offset estimation (FOE) algorithms, i.e., the radius directed fourth power algorithm and generalized circular harmonic expansion algorithm, were proposed to blindly estimate the frequency offset in the PS QAM coherent optical communication systems more accurately^[9]. For the sub-optimal issue of the blind phase search algorithm in the PS QAM signals, a supervised phase search has been proposed by estimating mean square error in the first stage to yield a noise rejection window^[10]. In addition, blind radius directed equalizer (RDE) with likelihood-based selection was proposed to improve the tolerance to the impairments caused by polarization mode dispersion^[11]. In the abovementioned DSP algorithms, blind equalization techniques are significant to compensate the signals distortion. The conventional blind dynamic channel equalization algorithms, such as cascaded multi-modulus algorithm (CMMA) and RDE, have the issue of amplitude dependence, as mentioned in Ref. [12]. Meanwhile, the probabilistic shaping with low entropy slightly affects standard amplitude radius and amplitude distribution. Hence, the fluctuating amplitude distribution caused by the imperfection of the transceiver, such as in-phase and quadrature (IQ) amplitude imbalance, IQ phase imbalance, and IQ skew, always degrades the performance of equalization algorithms.

In this paper, we propose a peak-density K-means clustering algorithm to mitigate the effect of signals’ amplitude variation on blind channel equalization. Here, the number of clusters, i.e., the number of the high-order QAM signals’ amplitude levels, is first identified by calculating the peak density and tagging the pre-selected centroids of each cluster in the signals’ amplitudes after clock recovery. Then, the K-means algorithm exploits the tagged centroids and the number of clusters to further realize clustering of signals’ amplitude levels. Assisted by the yielded cluster centroids and the classification boundary of signals’ amplitude levels, the optimized blind equalization algorithms can yield the accurate amplitude radius and amplitude distribution for superior performance. We experimentally demonstrated the proposed method in the PS 256-QAM coherent optical transmission over 80 km standard single-mode fiber (SSMF) and achieved approximately 1 dB optical signal-to-noise ratio (OSNR) improvement at the bit error rate (BER) of 1×10−3.

Blind dynamic channel equalization is indispensable in coherent optical communication to compensate various impairments, and the constant modulus algorithm (CMA) is widely utilized for quadrature phase shift keying (QPSK) signals. Figure 1(a) shows the butterfly structure of the CMA with four multi-tap finite impulse response (FIR) filters to include the interaction of multiple adjacent symbols. For high-order QAM signals, multiple reference amplitude radii are introduced to reduce residual errors, commonly used in the RDE and CMMA, as shown in Fig. 1(b).

The outputs of equalizer EX,Y in Fig. 1(a) can be obtained as^[13,14]$$\begin{gathered} EX(n)=hxx*(n)Ex(n)+hxy*(n)Ey(n), \\ EY(n)=hyx*(n)Ex(n)+hyy*(n)Ey(n), \end{gathered}$$(1)where (.)* represents the conjugate operation, and Ex,y is the N adjacent symbols prior to the nth input symbol. The row vectors hxx, hxy, hyx, hyy containing the N tap weights are optimized based on the stochastic gradient descent algorithm, given by $$\begin{gathered} hxx(n+1)=hxx(n)+µεx(n)EX(n)Ex*(n), \\ hxy(n+1)=hxy(n)+µεx(n)EX(n)Ey*(n), \\ hyx(n+1)=hyx(n)+µεy(n)EY(n)Ex*(n), \\ hyy(n+1)=hyy(n)+μεy(n)EY(n)Ey*(n), \end{gathered}$$(2)where μ is the adaptation step. The errors εx,y of RDE and CMMA are defined as $$\begin{gathered} εx,y−RDE=Rk2(n)−|EX,Y(n)|2, \\ εx,y−CMMA=(|||EX,Y(n)|−Rref1|−Rref2|−Rref3) \\ ·sign(EX,Y(n))·sign(Rref1−|EX,Y(n)|) \\ ·sign(Rref2−|Rref1−|EX,Y(n)||) \\ ·sign(Rref3−|Rref2−|Rref1−|EX,Y(n)|||), \end{gathered}$$(3)where sign is the symbolic function, and |·| represents the modulo operation. Taking the 16-QAM signal as an example, Rk is the standard amplitude radius, denoted as R1, R2 and R3 in Fig. 1(b). Rref in Eq. (3) is defined as $$\begin{gathered} Rref1=(R1+R2)/2, \\ Rref2=(R3−R1)/2, \\ Rref3=(R3−R2)/2. \end{gathered}$$(4)

The error of RDE is akin to the Euclidean distance between the data and the standard amplitude radius Rk, while the error of CMMA is akin to the multi-level referenced amplitude radius Rref. Both depend on the accurate measurement of the amplitude radius. For example, the decision of the amplitude is defined as (Rk−1+Rk)/2 in the RDE, as shown by red dash-dotted curves in Fig. 1(b), where D is described as the decision region of the amplitude. However, the signal distortion and noise will muddle up the amplitude distribution, and it is challenging to find an accurate standard amplitude radius and referenced amplitude radius in such an indistinct amplitude diagram.

For high-order PS QAM signals, the abovementioned issue becomes more critical. Fortunately, the symbols with lower energy are robust to various noises, and, therefore, the inner amplitude distribution in the amplitude diagram is more distinct. Thus, multiple inner amplitude rings can be extracted for error feedback to improve accuracy of the equalizer. However, probabilistic shaping makes more data concentrated in the interior, which causes more uneven amplitude distribution, and the standard amplitude radius will also have a slight offset. Besides, the standard amplitude radius is also influenced by the addition of pilot symbols and the transceiver imperfection. The pilot symbols with lower-order modulation will reduce the information entropy of probability shaping signal, thus slightly changing the standard amplitude radius and amplitude distribution^[12]. The imperfection of the transceiver including the IQ amplitude imbalance, IQ phase imbalance, and bias drifting of the modulator will also change the standard amplitude radius.

In order to enhance the robustness of the conventional blind equalizers for the PS high-order QAM signals and mitigate the effect of indistinct amplitude distribution caused by various degradations, we utilize a peak-density K-means clustering algorithm to find the accurate amplitude radii and implement the precise amplitude decision. Here, the peak density is first utilized to find the number of clusters (i.e., the number of amplitude levels) and cluster centroids (i.e., the standard amplitude radii). Assisted by the above information, the K-means clustering algorithm can easily escape from the dilemma of local optimization with a fast convergence to realize the optimum amplitude decision^[15].

Figure 2 illustrates the extracted partial amplitudes of the PS 256-QAM signal to explain the implementation of the optimized blind equalization. First, five innermost amplitude rings are extracted from the amplitude diagram of the PS 256-QAM signal after clock recovery, as shown in Fig. 2(a). Next, the amplitude information is shown in a polar coordinate system for easy clustering, as shown in Fig. 2(b). Then, the peak-density clustering is utilized to calculate the number of amplitude levels and the standard amplitude radii by finding the number of clusters and cluster centroids^[16].

Here, the key is that the cluster centroid of each cluster is always surrounded by points with lower local density, so the local density of the cluster centroid is the largest in this cluster^[17]. Thus, the local density ρ of each point is yielded based on the Gaussian kernel function, ρi=∑j≠ie−(disti,j/Dc)2.(5)

X=[X1,X2,…,XN] represents the amplitude points of N data symbols, i,j∈N. disti,j refers to the Euclidean distance between point Xi and point Xj, and Dc is the cutoff distance. If the distance disti,j is less than Dc, the point Xj belongs to the cluster of point Xi; otherwise, it is classified into other clusters. Next, all points are arranged according to the local density in descending order, and the data point Xi(i∈N) with the local density of ρi has been surrounded by i−1 points with higher local density.

In order to distinguish and extract the points with the peak density as the cluster centroids, the minimum distance δ between each point and the point with higher local density is calculated by δi=minj=1:i−1(disti,j).(6)

For the point Xl with the highest local density, the minimum distance is set to the maximum distance δl between point Xl and other points, defined as δl=maxj=2:N(disti,j).(7)

According to the decision diagram of local density ρ and relative distance δ, the number of amplitude levels can be obtained, as shown by red circles in Fig. 3. Thus, the required standard amplitude radii in the blind equalizer have been achieved.

After that, we need to yield the multi-level referenced amplitude radii and make an optimized decision of amplitude level in an indistinct amplitude diagram. With the assistance of the achieved number of the clusters and cluster centroids, K-means clustering is implemented to find the approximate boundaries of each cluster.

Figure 4 is the experimental setup of the PS 256-QAM coherent optical transmission system. At the transmitter, the probabilistic shaping is implemented based on the probabilistic amplitude shaping (PAS) in Ref. [18]. The pseudo-random binary sequence (PRBS) is firstly encoded by a constant component distribution matcher (CCDM) to convert a uniformly distributed sequence into a desired non-uniformly distributed sequence, and its output is used as amplitude bits^[19]. Then, the non-uniform amplitude bits are mapped into pulse amplitude modulation (PAM) 16 signals.

After that, the symbol sequence is up-sampled to four samples per symbol, and then we apply a root-raised-cosine (RRC) finite impulse response filter with 0.35 roll-off factor to mitigate the impact of inter-symbol interference (ISI). Next, the auxiliary symbols are appended for clock timing and FOE. Two series of PAM-16 signals are imported into the 8 GSa/s arbitrary waveform generator for digital-to-analog conversion and modulated by a Mach–Zehnder modulator to generate PS 256-QAM signals. The modulated 256-QAM signals are launched into the 80 km SSMF and received by the coherent optical receiver. The first variable optical attenuator (VOA) is used to adjust the input power to the fiber, and the second one is utilized to vary the OSNR for BER measurement. At the receiver, the PS 256-QAM signals are acquired by a 50 GSa/s real-time oscilloscope for analog-to-digital conversion. Then, the off-line DSP algorithm implements the operations of resampling, RRC filtering, and IQ orthogonalization. After that, the chromatic dispersion of fiber is compensated in the frequency domain. Next, clock recovery is implemented based on the G-TED algorithm with auxiliary symbols. After that, the proposed optimized blind equalization of CMMA and RDE with 33 filter taps is applied for channel equalization. Afterwards, accurate frequency offset correction is carried out, and then 20 pilot symbols are utilized for initial phase correction before completing carrier phase estimation (CPE). Furthermore, in order to mitigate the nonlinear distortion caused by optoelectronic devices, the second-order Volterra nonlinear equalization (VNLE) is exploited for both in-phase and quadrature components of the symbols separately, which causes slight phase shifts. Due to the phase-sensitive characteristic, a 121-tap direct-decision least mean square (DD-LMS) equalizer is used to compensate residual linear impairments and slight phase shifts prior to the symbol decision.

In this work, we verify the feasibility of the proposed method and evaluate its performance based on the 7.4 bit/symbol 256-QAM signal in Fig. 4, where the equalization performance is significantly affected by the non-uniform amplitude distribution. However, the clock recovery is effective, which is more conducive to the identification of the inner amplitude levels.

For the extracted amplitudes from the amplitude diagram of the PS 256-QAM signal with 7.4 bit/symbol in Fig. 5, we calculate the standard amplitude radii, as shown by purple crosses, and the corresponding decision region of amplitudes is plotted by red lines. Here, the extracted amplitude diagram of the polar coordinate in Fig. 2 is overlapped into a small interval in order to clearly illustrate the amplitude variation. Based on the proposed peak-density K-means clustering method, the amplitude radii and the decision region of amplitudes are optimized for CMMA and RDE equalization, as shown by black crosses and black lines. It is obvious that the standard radii and decision region of amplitude have drifted slightly.

Figure 6 shows the measured BER versus OSNR of a 7.4 bit/symbol 256-QAM signal over 80 km SSMF with the blind equalizers of standard-CMMA (STD-CMMA), standard-RDE (STD-RDE), and the modified CMMA and RDE with the proposed method. The inset is the constellation diagram of the signal with the OSNR of 26.36 dB. Compared with the STD-CMMA and STD-RDE equalizers, the modified RDE and CMMA in this work both have an improvement of approximately 1 dB at the 1×10−3 BER. In practice, it is challenging for CMMA to define excessive referenced radii due to high computational complexity in Eq. (3). On the contrary, the judgment condition is simple in the RDE. The internal multiple referenced radii are only adjusted, and the other referenced radii remain unchanged, which guarantees the small convergence error. As a result, the modified RDE outperforms slightly the modified CMMA, similar to the standard algorithms, as shown by pentagram-marked curves and circle-marked curves in Fig. 6.

In this work, we proposed and experimentally demonstrated a peak-density K-means algorithm to optimize blind equalizers of RDE and CMMA. We have successfully verified the feasibility of the proposed method in a PS 256-QAM coherent optical transmission system and achieved approximately 1 dB OSNR improvement for the 7.4 bit/symbol 256-QAM signal’s transmission over 80 km SSMF at the 1×10−3 BER. The proposed method mitigates the dependence of blind equalization on amplitude variation, which is promising for PS high-order QAM coherent optical communication.