With the theory of the modified color difference saliency algorithm as a base, the fabric image texture clearance regions are characterized by the area and roundness of the highlighted and significantly connected regions. By the statistical extraction of the distribution laws of the standard fabric image texture clearance and by the setup of the color difference rating standard, not only the big serious color difference region is quickly detected, but also the background color noise of the fabric texture clearance region of the fabrics without the big serious color difference is distinguished. Based on the color difference rating standard, the weighted average method is used to complete the tested fabric color difference grating for the fabric images beyond the fabric texture clearance region. The results show that the proposed algorithm has the minimum error mean and variance, which can reflect the true color difference of fabrics.