With the increasing global demand for high-speed satellite communication, 100 Gb/s intersatellite coherent optical communication technology is becoming the main solution for efficient intersatellite data transmission. The application of coherent optical communication significantly improves the data transmission rate, can satisfy the requirements for large-capacity data transmission, and is suitable for communication between low Earth orbit. Intersatellite coherent optical communication technology, combined with digital signal processing (DSP) algorithm, reduces bit error rate (BER) and ensures reliable transmission via effective channel equalization and signal demodulation. In this study, high order modulation scheme combined with DSP is used to verify 100 Gb/s link transmission performance, and the influence of satellite pointing error and inter-satellite distance on transmission performance is analyzed. The simulation results show that quadrature phase shift keying (QPSK) exhibits stronger robustness to satellite pointing error and inter-satellite distance than 16 quadrature amplitude modulation (16-QAM) signal under 100 Gb/s transmission. At a BER of 3.8×10^-3, the pointing error and intersatellite distance tolerance of QPSK are 1.3 and 1.8 times those of 16-QAM signal, respectively. The spectrum efficiency of 16-QAM signal is higher, which can reduce the requirements of bandwidth and sampling rate of photoelectric devices. Therefore, in a future 100 Gb/s inter-satellite coherent optical communication system, the device bandwidth and satellite dynamic characteristics can be comprehensively considered to complete the system design.