In order to minimize the error, the non-lens imaging method is applied in the interferometric measurement of point-diffraction spherical wavefront precision. In the case of high numerical aperture and large lateral displacement, significant additional systematic error can be introduced into point-diffraction wavefront measurement result, and it cannot be removed with traditional calibration method. A high precision calibration method of structure error for point-diffraction wavefront interferometric measurement based on three-dimensional coordinate reconstruction and symmetric displacement compensation is proposed to calibrate the systematic error introduced by high numerical aperture and large lateral displacement in the measurement of point-diffraction spherical wavefront with non-lens imaging. Firstly, three-dimensional reconstruction method is used to pre-calibrate the symmetric error. Then, because the error introduced by point source lateral displacement in interferometric measurement is symmetric, the symmetric lateral displacement compensation is applied to further correct the residual structure error due to three-dimensional coordinate reconstruction error. Both the numerical simulation and experimental measurements are carried out to verify the feasibility of the proposed calibration method. According to the analyzing results, the calibration method can reach the correction precision better than λ/10000. The proposed calibration method is of great significance for the application of high precision calibration of structure error introduced by non-lens imaging interferometric measurement and high-accuracy measurement calibration of point-diffraction spherical wavefront.