A method to compensate the orthogonality deviation of the Moire fringe photoelectric signal was proposed to improve the accuracy of a photoelectric rotary encoder. Based on Hilbert transform theory, a dynamic measurement algorithm for the orthogonality deviation of same frequency photoelectric signal was constructed. In the light of the mathematical model for Moire fringe photoelectric signal, the influence of orthogonality deviation on subdivision precision was analyzed and an angle compensation model was set up. According to the working principle of the encoder, a synchronous processing mode was adopted to compensate the photoelectric signal and update dynamically the lookup table for angle codes simultaneously; by the switching of subdivision lookup table, the orthogonality deviation for the photoelectric signal was compensated in real time. Finally, a 23-bit photoelectric encoder with 18° orthogonality deviation was compensated using this method. Experimental results show that the peak value of subdivision error of the encoder has reduced from 4.79" to 1.26 " after compensation. This method can actually be used in an encoder system and it can improve the subdivision accuracy, environmental adaptability and the reliability of encoders.