This study aims to enhance the self-calibration accuracy of a binary polarization-state analyzer (PSA) by proposing a multivariable optimization method. This study is motivated by the limitations of traditional PSAs, which are highly susceptible to errors induced by optical power fluctuations, temperature variations, and wavelength dependencies. These factors significantly affect the measurement accuracy of state of polarization (SOP). To address these challenges, this study proposes an optimization method that resolves the self-calibration problem and improves the accuracy of SOP analysis. The primary objective is to minimize the influence of measurement errors and noise, thereby enhancing the accuracy and reliability of the system. Additionally, this study introduces a trust region reflective algorithm to efficiently solve the multivariable optimization problem, and its effectiveness is verified by Monte Carlo simulations and sensitivity analysis.

This study proposed a mathematical model for a 4-bit binary PSA, formulated as an overdetermined system of equations to characterize both the self-calibration process and SOP analysis. The model contains indispensable parameters, including the rotation angles of the magneto-optic (MO) rotators, retardation angle of the quarter-wave plate (QWP), and relative orientation between the QWP and polarizer. The proposed multivariable optimization model utilizes the least-squares method in matrix form with an objective function designed to minimize the optical power residual errors and degree of polarization (DOP). To solve this optimization problem, a trust region reflective algorithm was employed, providing an iterative refinement mechanism for the system parameters. This algorithm ensures efficient convergence by constraining the variables within specified bounds, thereby preventing unreasonable solutions. Furthermore, a one-at-a-time (OAT) method was applied to the sensitivity analysis to assess the influence of various SOPs on the optimization outcomes. Monte Carlo simulations were also conducted to simulate diverse operational conditions and evaluate the accuracy of both the self-calibration and SOP analyses.

The proposed optimization method was evaluated through simulations and experimental validations. The simulation results demonstrate significant improvements in both self-calibration and SOP analysis accuracy compared to the direct method. For self-calibration, the accuracies achieved for parameters φ, θ_p, and Γ are recorded as ±0.1407°, ±0.2106°, and ±0.3506°, respectively. Additionally, the optimization approach reduces the SOP measurement errors and improves the DOP accuracy compared with the direct method. Sensitivity analysis reveals that the performance of the method is highly dependent on the SOP for self-calibration, with certain regions of the SOP yielding faster convergence and higher accuracy. Monte Carlo simulations further highlight the robustness of the method under varying noise conditions, confirming its superior performance relative to the direct method. In the experiment, a 4-bit binary PSA was constructed using a tunable laser source (TLS) and a polarization controller (PC) to generate light with arbitrary SOPs. Experimental results demonstrate narrow measurement distributions for φ, θ_p, and Γ, with self-calibration accuracies for these parameters closely aligning with the simulation results. Specifically, the measured values for φ, θ_p, and Γ are recorded as 22.9504°±0.1407°, 89.2219°±0.2106°, and 92.9386°±0.3506°, respectively. The SOP (S₁, S₂, S₃) errors are substantially reduced, and the DOP error is found to be less than 0.004%. These data indicate that the optimization method can provide more accurate and reliable SOP measurements.

In conclusion, the multivariable optimization method proposed in this study proves to be a highly effective solution for improving the self-calibration and accuracy of SOP analysis in binary PSA. The method successfully addresses challenges related to optical power noise, temperature variations, and wavelength dependencies, offering a robust framework for achieving high-precision measurements. The results from both simulations and experiments demonstrate that the proposed optimization method achieves a higher accuracy than the direct method, and the self-calibration errors of φ, θ_p, and Γ could be reduced to 0.1407°, 0.2106° and 0.3506°, respectively, and SOP and DOP errors could be reduced to 1/2.8 and 1/1922.5 of their original values, respectively. Moreover, the wavelength-dependent coefficients of φ and Γ are found to be -0.0277 (°)/nm and 0.5694 (°)/nm, respectively. These results suggest that the proposed optimization method can be applied to other optical measurement systems to address similar calibration challenges. In the future, Lu–Chipman decomposition will be combined with the proposed method to self-calibrate the diattenuation, retardance, and depolarization characteristics to further improve the accuracy of the SOP analysis. This study provides a foundational framework for advancing SOP analysis techniques with broader implications for optical measurement systems in a range of scientific and industrial applications.