In structured light 3D measurement technology, the system calibration is the basis of measurement. A method was introduced that projected a coding grating onto a checkerboard calibration plate and calculated its projected pixel coordinates from the phase values at the corner points. Considering the problems of phase missing and phase anomalies in the corner neighborhood, a fitting algorithm based on local random sample consensus (RANSAC) was proposed to eliminate phase outliers and surface fitting interpolation, thereby calculating the sub-pixel level projected pixel coordinates of the feature points for calibration. This method does not rely on the calibration results of the camera and is also applicable to the dot calibration plate, effectively filtering the anomalies and noise of the phase values near the feature points. Experimental results show that this method has good robustness to outliers, and the reprojection error reaches 0.09 pixels, which has lower requirements on the type and cost of calibration plates, and has certain practical values.