To making the high resolution grating, a numerical calculation was used to analyze the effect of recording parameters on groove density, focal curve and imaging performance of the grating and their compensation. Based on Fermat’s principle, light path function and aberration, the effect on imaging performance of the grating was analyzed. In the case of fixed using parameters, the error of the recording angle has a greater influence on imaging performance, therefore the gain of the weight of recording angle can improve the accuracy of the recording angle values in the optimization; recording distance has little influence on imaging performance; the relative errors of recording parameters cause the change of imaging performance of the grating; the results indicate that recording parameter errors can be compensated by adjusting its corresponding parameter. The study can give theoretical guidance to the fabrication for high resolution varied-line-space plane holographic grating in on-line spectral diagnostic and reduce the alignment difficulty by analyze the main error effect the imaging performance and propose the compensation method.