With the rapid development of optical communication technology toward high capacity, large bandwidth, and high speed, the multi-dimensional multiplexing technology is widely researched and adopted. Polarization multiplexing technology is an important multiplexing technique. However, polarization introduces damage to polarization multiplexing systems. In extreme weather conditions such as lightning near optical cables, the Kerr effect, and the Faraday effect, rapid rotation of the polarization state of the signal can be caused. This rotation disrupts the orthogonality of the two polarization states, thus increasing the bit error rate. Therefore, it is significant to trace and compensate for polarization state rotation. Currently, equalization algorithms for rotation of the state of polarization (RSOP) include the constant modulus algorithm (CMA), the Kalman filtering algorithm, and its derivative algorithms. The CMA is simple to implement but becomes ineffective when RSOP changes rapidly. In recent years, the focus has been realized by the Kalman filter and its derivative algorithms, including the extended Kalman filter (EKF), covariance Kalman filter (CKF), and square root covariance Kalman filter (SCKF). The EKF yields high tracking and compensation accuracy for RSOP but requires the calculation of the Jacobian determinant, which results in high algorithm complexity. The CKF avoids the computation of the Jacobian determinant, significantly reducing algorithm complexity. Although the SCKF avoids the positive definite decomposition of the state error covariance matrix in CKF, during the adaptive SCKF implementation, the process noise matrix still needs to calculate out positive definite decomposition, which cannot be fully guaranteed during the actual algorithm execution. We propose a new RSOP equalization algorithm based on adaptive square root cubature Kalman filtering. This algorithm avoids the positive definite decomposition of Q and exhibits adaptive updating of the noise covariance matrix in various scenarios, thus enhancing the algorithm’s robustness.

A residual decision-adaptive square root cubature Kalman filtering based on the square root of Q(RD-ASCKF-SQ) algorithm is proposed for equalizing RSOP. This algorithm initiates a time update by calculating cubature points using the square root matrix of the state error covariance from the k-1 time or the initialized state parameter prediction. Subsequently, cubature points after the state transition are computed based on the state transition function to obtain the predicted state vector and the predicted square root of the state error covariance for the current time k. The next step is to proceed to a measurement update. The new cubature point set is calculated again from the state prediction in the previous step. Then the propagation cubature point is calculated according to the measurement transfer equation. Meanwhile, the predicted measurement values and the square root of the innovation covariance matrix at time k are calculated. Finally, by computing the square root of the self-covariance and cross-covariance matrices, the Kalman gain is obtained. The innovation is then calculated based on the error between the actual and predicted measurements. Additionally, combining the Kalman gain for signal recovery can yield the final state estimation, and residual decision detection can help decide whether the process noise Q should be updated.

We conduct numerical simulations on a 112 Gbit/s PDM-QPSK system to validate the performance of the RD-ASCKF-SQ algorithm. Meanwhile, we perform simulation analyses to assess the performance differences between ACKF and RD-ASCKF-SQ under varying rates of RSOP changes. For RSOP azimuthal angle change rates ranging from 10 Mrad/s to 120 Mrad/s, the average bit error rate of RD-ASCKF-SQ is lower than that of ACKF. Additionally, the bit error rates of RD-ASCKF-SQ at different RSOP change rates all meet the 7% forward error correction threshold. Simulation analyses also examine the bit error rate curves of SCKF and RD-ASCKF-SQ under different signal-to-noise ratios (SNRs). In the statement of RSOP azimuthal angle change rate of 40 Mrad/s and low SNR, the SCKF algorithm fails to converge when the diagonal elements of Q are set to 10^-2 and 10^-4, and it achieves convergence only when the values are set to 10^-6. In contrast, RD-ASCKF-SQ converges with Q diagonal elements set to 10^-2, 10^-4, and 10^-6. RD-ASCKF-SQ exhibits greater tolerance to different initial Q values than SCKF, converging adaptively to appropriate values for RSOP equalization. The introduction of square root coefficients is analyzed for its influence on the convergence probability and average bit error rate of the ASCKF algorithm. Before the introduction of square root coefficients, ASCKF yields a convergence probability of 95% at RSOP azimuthal angle change rate of 70 Mrad/s and 94% at 80 Mrad/s. In contrast, RD-ASCKF-SQ consistently achieves a convergence probability of 99%. RD-ASCKF-SQ avoids the positive definite decomposition of Q, reducing algorithm complexity and further enhancing stability. Furthermore, we analyze the adaptive update counts of ASCKF and RD-ASCKF-SQ for RSOP azimuthal angle change rates ranging from 10 Mrad/s to 120 Mrad/s. RD-ASCKF-SQ exhibits fewer update counts at lower RSOP values, gradually increasing as RSOP values rise. Meanwhile, traditional ASCKF requires adaptive updates at every time step, and the update counts are independent of the algorithm’s convergence status. Compared to traditional ASCKF, RD-ASCKF-SQ reduces unnecessary adaptive updates, thereby improving algorithm runtime speed.

We propose an RSOP equalization algorithm based on RD-ASCKF-SQ. The basic idea of this scheme is to update the square root of the error covariance matrix directly by the square root coefficient. It avoids the positive definite decomposition of Q in each adaptive process. Additionally, it combines a residual decision detector to impose constraints on parameter updates. The algorithm updates Q when it diverges and stops updating when it converges, thereby improving running speed and reducing running time. The algorithm performance is validated by numerical simulations in a 112 Gbit/s PDM-QPSK system. The proposed algorithm demonstrates adaptive capabilities, showing higher robustness under improper Q selection than the SCKF algorithm. Additionally, compared to the ASCKF algorithm without square root coefficients, it exhibits superior stability. These characteristics hold across different scenarios, highlighting the algorithm’s commendable generalization performance.