By modeling and analyzing on the carrier recovery based on COSTAS optical phase locked loop, we obtain the closed-loop transfer function, error function and carrier signal that to be recovered of the system. Then we design composite control to realize the fast and small range tracking by using the acousto optic frequency shifter in the inner ring, and quickly pull into capture tape by control laser in the outer ring. In order to get the good carrier recovery and signal demodulation, we design the phase detection module based on exclusive-OR gate (XOR) to analyze the phase detector. The phase detection ranges from -42 ° to +42 °, and phase detector gain achieves 14.4 mV/(°) when the loop bandwidth is 1.5 MHz. The experimental results show and that the carrier recovery and signal demodulation is good when the code rate is 5 Gb/s, and the error rate is 1.55×10 -8 when the signal power reaches -40.4 dBm. With the rate increasing or decreasing, the system performance decreases, but the system can recover the carrier signal and demodulate the data. This system can provide references for homodyne coherent communication laboratory verification.