In order to improve the reliability and repeatability of phase counting and subdivision of quadrature signals for homodyne interferometers, a high accuracy error compensation algorithm for quadrature signals combined error correction with phase-based counting and subdivision was proposed based on the Heydemann correction model of quadrature signals and digital signal processing technology. Matrix manipulation based on the least square method was adopted to calculate compensation parameter initials of quadrature signals, which were corrected by iterative operation to further improve the compensation accuracy. Then phase-based counting and subdivision algorithm was established base on the revised signals. Finally, the algorithm was verified with Matlab software. Experimental results show that the proposed algorithm can accurately compensate the error of the quadrature signals, and the measurement accuracy achieves sub-nanometer or even picometer scales, thus the demodulation accuracy of the measured signals can be improved effectively.