A semiglobal stereo matching algorithm based on fusion cost and adaptive penalty coefficient is proposed to address the low-accuracy problem of semi-global stereo matching algorithms in discontinuous disparity regions. In the cost calculation part, a fusion cost calculation method is proposed introducing the gradient in the y direction of the input image and combining the fusion formula with the gradient in the x direction of the input image, absolute difference, and Census transform to form the cost calculation data item; in the cost aggregation part, a pixel classification mechanism is proposed that classifies each pixel by color and gradient dual thresholds and adaptively adjusts the size of its penalty coefficients; finally, the initial parallax map is processed using a multistep parallax optimization method. Results show that the average error of the proposed algorithm in the discontinuous parallax regions decreases by 1.1 percentage points to 12.8 percentage points, and it also decreases in non-occlusion and all regions. The proposed algorithm exhibits high matching accuracy and robustness.