In order to solve the problem that traditional Chan-Vese (CV) model is difficult to extract metallographic grains quickly and accurately, the metallographic image segmentation method based on improved CV model integrated with local fitting term is proposed. We use the reciprocal cross entropy threshold segmentation rule to replace the regional term of the energy function in the traditional CV model and construct a new level set model. The proposed model can minimize the reciprocal cross entropy between original and segmented image, and accurately segment the metallographic images with more noises and larger local gray scale. In addition, Taking that the reciprocal cross entropy will increase algorithm’s computational complexity into account, the maximum absolute median difference is adopted to adjust energy weight inside and outside the curve to accelerate curve evolution. The distance regularized term is introduced to avoid initialing level set function, and accelerate the model convergence. Experimental results show that comparing with other traditional CV models, the proposed model has obvious advantages both in segmentation result and efficiency.