In order to make the received two signals orthogonal to each other and correctly demodulate the signal, Schmitt orthogonalization algorithm was used to theoretically analyze and experimentally verify the two signals. The performance of constellation was improved after orthogonal imbalance compensation. The carrier frequency was tracked and captured by the loose tail ring to achieve carrier recovery, and the baseband signal was demodulated correctly. The experimental results show that, after compensation by Schmidt orthogonalization algorithm, the rotated constellation is corrected and the problem of unequal Euclidean distance between constellation points is improved. The baseband signal distortion becomes corrected. When the frequency offset is in the range of -300kHz~300kHz, the loose tail loop can track and capture carrier frequency and baseband signal can be correctly demodulated. The scheme is low complexity and practical, and is suitable for the study of coherent optical communication systems.