Laser communication has outstanding advantages such as large information capacity, high optical gain, high anti-interference and anti-interception capabilities. It was an important technical means to solve high-speed communication problems. According to the requirements of transceiver isolation and two-way duplex working mode in laser communication. After simulating and analyzing the isolation of different optical structures, it was proposed to use off-axis three-mirror optical antennas to reduce backscattering to achieve high transceiver isolation. In the optical design, the smallest incident angle on the optical surface was controlled through iterative optimization, and clear index requirements were put forward for the fine processing technology of the optical surface, and finally the isolation was more than 70 dB. The simulation analysis shows that the object field angle of 5 mrad isolation is 73.7 dB, and the measured isolation of Φ150 mm off-axis three-mirror optical antenna can reach 73 dB, which is consistent with the results of simulation analysis. It meets the requirements of satellite laser communication system tracking and communication for light antenna isolation, and can be used for inter-satellite laser communication.