This study proposes a surface defect detection method based on the scattering field distribution fitting approximation to accurately detect different sizes of optical element surfaces, especially small-size defects, with typical optical element surface defects-pits and scratches-as the research target. Experimental results show that the method can rapidly and effectively detect small-size defects on optical element surfaces and the relative error between fitting calculation results and the original size of a sample is basically less than 5%, which verifies the effectiveness of the method. In addition, the method addresses the problems of low accuracy and complex structure of existing measurement methods and introduces a new idea for accurately detecting microsized defects on optical element surfaces.