In order to improve the reconstruction effect of digital holographic microscopy system, the Project Onto Convex Set (POCS) algorithm is applied to digital holography to achieve super-resolution reconstruction of hologram. Firstly, based on the theory of digital holography and POCS algorithm, the optical path of transmissive off-axis digital holography is established. Then, the resolution board is used to verify the effectiveness of the POCS algorithm in hologram super-resolution restoration. Finally, the feasibility of the method in 3D reconstruction is verified by microlens array. The residual estimation results of the low-resolution hologram and the super-resolution restored hologram are respectively estimated by the three-dimensional reconstruction result of the reference hologram. The PV and RMS of the latter residuals are reduced by 36.88% and 70.66%, respectively, which proves that the method can achieve super-resolution reconstruction of digital holography.