Camera calibration is significant in machine vision and is widely applied to 3D reconstruction, defect detection, visual navigation, etc. To improve the calibration result accuracy for intrinsic and extrinsic parameters, we propose a camera calibration optimization method based on the adaptive extended Kalman filter (AEKF) algorithm. Zhang's calibration method based on a 2D plane target is a commonly adopted camera calibration approach. Kalman filter (KF), extended Kalman filter (EKF), and unscented Kalman filter (UKF) have been introduced to further enhance the accuracy of Zhang's calibration method. The predicted value of the previous moment and observation value of the current moment are employed to accurately predict the state vector, providing an efficient and precise method to estimate the camera calibration state. EKF algorithm linearizes the nonlinear state equation by performing a first-order Taylor expansion of the nonlinear function and neglecting the other higher-order terms. Some scholars have applied the EKF algorithm to the camera calibration and yielded better calibration results than Zhang's calibration method. The introduction of a state estimation method can improve the camera calibration accuracy. However, the initial parameter setting of process and observation noises in the EKF algorithm, which affects the optimization of the camera calibration parameters, greatly depends on the user's judgment and choice, and has certain limitations and poor robustness in noisy environments. Therefore, we want to propose a method to perform the EKF-based camera calibration method without dependence on the initial parameter setting, update the process and observation noise covariance matrices employing the innovation between the predicted and observed values, and exhibit good robustness in noisy environments.

EKF cannot automatically select and adjust the process and observation noises in the camera calibration, which makes the camera calibration accuracy overly dependent on the user's judgment and inputs of the initial parameters. Thus, the innovation between the predicted and observed values is utilized to update the process and observation noise covariance matrices to adaptively adjust the variation of the process and measurement noises. To address the problems of existing methods, we build a camera projection model based on the imaging principle of the lens and develop an adaptive innovation-based EKF camera optimization calibration method. The unit quaternion is adopted to represent the rotation matrix, the intrinsic and extrinsic parameters of the camera are the state vectors, and the image coordinates of the detected feature points on the two-dimensional checkerboard target are the observation vectors to build the process and measurement model of the AEKF algorithm respectively. The extracted feature points are filtered point by point to obtain the optimal estimation of intrinsic and extrinsic parameters of the camera, and the process and observation noise covariance matrices are updated during the iterative process with the change of the innovation. Meanwhile, the reprojection error is utilized to assess the optimization algorithm performance, and different noise levels are added to validate the algorithm robustness. The EKF-based camera calibration optimization method is introduced to solve the problems that nonlinear filtering depends on the initial parameter setting, the fixed initial parameter is unfavorable to the filtering process under noise changes, and the EKF has poor robustness in noisy environments.

The process and observation noises in the captured images vary during the actual calibration. To overcome the limitation of EKF's inability to adaptively adjust the process and observation noises in camera calibration, we design the innovation between predicted and observed values to update the process and observation noise covariance matrices. AEKF algorithm is presented to optimize the intrinsic and extrinsic parameters of the camera, becoming more suitable for actual applications and eliminating the reliance on fixed initial values for the process and observation noises set by human interventions. A virtual camera and a virtual checkerboard target are constructed based on the camera model. The intrinsic and extrinsic parameters of the virtual camera (state vector) and the 2D image coordinates of the feature points (observation vector) are obtained. Additionally, the reprojected error of the proposed AEKF algorithm is lower than that of other methods (Table 1), which improves calibration accuracy for the virtual camera. The experiments are carried out using a USB camera and an industrial camera respectively. The optimized calibration results of the AEKF algorithm exhibit lower reprojection errors (Figs. 8 and 11) and demonstrate faster convergence and smaller oscillations during the iterative process. The proposed AEKF algorithm still has low reprojection error in the case of gradually increasing noise, which indicates that it has high robustness (Figs. 9 and 12). The effectiveness of the AEKF algorithm is verified by simulation and experiments. The calibration results obtained by the USB camera and industrial camera improve by 61.17% and 12.17% compared with Zhang's calibration method respectively. This algorithm outperforms UKF and EKF in noisy environments in calibration accuracy and robustness, making it applicable to various machine vision fields such as 3D reconstruction, visual navigation, robot localization, and defect detection.

The proposed AEKF algorithm modeled by the camera projection is employed to optimize the intrinsic and extrinsic parameters of camera calibration, which can improve the mapping accuracy between pixel coordinates and world coordinates. Experimental results demonstrate the effectiveness and feasibility of the AEKF algorithm, leading to a reduction in reprojection errors of camera calibration results. The process and observation noise covariance matrices are updated based on the innovation during the iteration process to eliminate the reliance on the user's judgment. The reprojection error of camera parameters using the AEKF algorithm is significantly lower than that of the EKF algorithm and Zhang's calibration method. Meanwhile, the reprojection error of the AEKF algorithm under the environment of gradually increasing noise is generally lower and grows slowly compared with that of UKF and EKF. Additionally, this algorithm has high accuracy and robustness and can enhance the accuracy of the calibration results, providing better assurance for tasks such as image processing, 3D reconstruction, pose estimation, and machine vision.