Based to the Lagrange invariant law, the lateral and axial imaging resolution of self-interference incoherent digital holography (SIDH) imaging system is discussed in detail, comparing with the resolution of classical optical imaging system. The mathematical expression for the lateral, axial imaging resolution and the resolution capability criterion are presented in theory, according to the system magnification and the full-width at half-maximum of the point spread function. It indicates that the SIDH can improve the lateral resolution of one time when the hologram is recorded at the position, and the two spherical waves from the object point are perfectly overlapped, comparing with the classical coherent optical imaging systems. The compression sensing numerical reconstruction algorithm is introduced to improve the axial resolution of system, and the corresponding numerical simulation and experimental results are obtained. The analysis results have important guiding significance for SIDH in imaging, measurement and optical system design.