The image background of the crack on the sagger bottom is complicated, the distribution of cracks is dense and intermittent, and characteristics of cracks are not obvious, so crack extraction of the sagger bottom is difficult. To solve the problems, a method for detecting sagger cracks based on sector neighborhood difference histogram is proposed. A multi-scale, multi-direction sector filter is constructed according to the spatial clustering characteristic and directional characteristic of the crack pixels. By calculating the convolution of the filters with the image, a sector neighborhood difference histogram that can reflect the crack distribution probability feature is obtained. Crack extraction is realized depending on the difference in crack distribution probability characteristics between the crack pixels and non-crack pixels. Finally, the global and local length and distribution area characteristics of the cracks are integrated to evaluate the degree of cracking. The experimental results show that the proposed algorithm can achieve good extraction results for all types of cracks on the sagger bottom. The precision and recall of the algorithm can reach higher than 90%, which is better than some of the existing good methods for crack extraction. The assessment results of the method for assessing the severity of cracks are also basically the same as those of a person's subjective assessment.