The high mortality rates of colon and rectal tumors have put forward an urgent need for rapid, sensitive, and accurate diagnosis. The polarization imaging technology, with the advantages of noninvasiveness, noncontact, quantification, rapidity, and high sensitivity, is expected to be used for auxiliary diagnosis of colorectal cancer. Herein, the differences in colorectal tissues of four pathological types were studied using this powerful technology. Polarized light imaging combined with the Mueller matrix decomposition (MMPD) method was applied to extract structural features that may be related to colorectal tumors. It demonstrated that parameters δ and θ could reflect the structural differences of colorectal tumors. Preliminary simulated experiment results revealed that the parameter δ was related to the fiber density, and the parameter θ was related to the fiber angle. Then Tamura image texture analysis was used to quantitatively describe tissues of different pathological types, and the results showed that the coarseness, contrast, directionality, and roughness of the four groups were statistically different. Texture analysis based on the quantitative data of the four dimensions could be applied for the identification of benign and malignant colorectal tumors.