PatchMatch-based algorithms that simultaneously estimate the disparities and normal unit of a disparity plane have achieved highly accurate sub-pixel disparities in the stereo matching problem; however, this kind of methods can not effectively deal with error matching in the non-texture regions of image. To solve this problem, we improve the LocalExp(local expansion move)algorithm and present a new stereo matching algorithm integrating multidimensional information for adaptive pixel category optimization. First, a crossover window is designed,the color and color self-correlation information in the window are used to establish the weight, and the restrained function is utilized to eliminate the outliers in the matching cost. Second,the constraint mechanism is added to the label initialization procedure, the proposal generation mechanism is modified, and the local expansion movement algorithm is used to optimize the label values. Finally, the pixel category information-based filling strategy is used to refine the disparity. The experimental results show that the proposed method can obtain a low matching error on the Middlebury dataset.