If Zhang’s camera calibration results are optimized with SBA directly, different sets of camera parameters (internal parameters and distortion parameters) will be obtained.Based on the mathematical model of SBA and the equality constraints of camera parameters, a Constrained Sparse Bundle Adjustment (CSBA) algorithm is proposed with a new block matrix partition strategy to improve the efficiency of solving sparse linear equations.Simulation experiments are implemented to verify that unified camera parameters can be obtained even if the pixel coordinates don’t have zero-mean Gaussian error.Finally, the CSBA algorithm is applied to a binocular stereo vision system.The experimental results demonstrate that the CSBA algorithm can optimize the camera parameters and position parameters simultaneously, and improve the accuracy of 3D reconstruction.