In this paper, we proposed an adaptive region fitting non-uniform image segmentation algorithm to solve the low segmentation accuracy of the traditional level set method in segmenting uneven grayscale images. First, we constructed an adaptive region fitting energy term to retain more detailed information in the local region of the image to be segmented, and to achieve accurate image segmentation. Second, we applied a non-convex regular term to smooth the curve and protect the edges of the image. Furthermore, we added an energy penalty term to constrain the level set function and improve the algorithm’s segmentation efficiency. Finally, the synthetic and real images were verified by experiments. Experimental results show that the average Dice similarity coefficient, Jaccard similarity coefficient, and accuracy of the proposed algorithm are 88.62%, 79.86%, and 92.48%, respectively, which are 18.19 percentage points, 16.10 percentage points, and 13 percentage points higher than those of Local Binary Fitting (LBF), Local and Global Intensity Fitting (LGIF), and Local Pre-fitting (LPF) respectively.