To meet the requirements of timeliness and reliability of the lunar rover system for obstacle detection, a semi-global stereo-matching algorithm based on the plane constraint and adaptive penalty coefficient was proposed. First, the Scale Invariant Feature Transform (SIFT) feature extraction and matching algorithm was used with two images of epipolar rectification from which edges are extracted. Second, the matched SIFT feature points were used to fit the plane, and the disparity search ranges of all pixels were estimated in the left and right images. Finally, based on the traditional semi-global matching algorithm, the adaptive penalty parameter strategy was applied to the stereo matching of the left and right images. Experimental results show that the algorithm can effectively reduce computational complexity to 19.9% that of the traditional methods. In addition, this method can obtain accurate matching results for the disparity discontinuous and occlusion regions. Compared with the traditional semi-global matching method, the processing speed and matching accuracy are clearly improved. This method can provide a foundation for further application.