A measuring error separation method was proposed on the basis of self-calibration algorithm for an imaging instrument. A self-calibration model was established and verified by the imaging instrument. A regular grid plate was used to replace the standard, and the grid plate was put on the image instrument stage in three different positions to get three sets of measurement data. The data were put into the self-calibration model through the developed transformation model in advance. Then, the system error contained in the measurement results was separated, and self-calibration was implemented. As measurement data transformation model was established, the problem of misalignment between grid plate and stage coordinate system was solves effectively during the process of experiment and the measuring accuracy of the imaging instrument was improved. Experiments show that the error of the measurement point is or millimeter scale without the self-calibration algorithm, but the maximum and minimum errors of the measurement point after self-calibration are 0.49 μm and 0.00 μm respectively. The results mean that it is feasible to apply the self-calibration algorithm to error separation of measurement results for the image measuring instrument.