When optical signals are transmitted through atmospheric turbulence channels, the signal transmission quality is degraded due to channel attenuation caused by aerosol particles in the atmosphere, as well as beam drift and scintillation due to atmospheric turbulence. This leads to an increase in the communication bit error rate and a decrease in channel capacity, which severely affects the performance of the communication system. Therefore, based on the real-time changes in the atmospheric channel state, an efficient adaptive transmission scheme is designed at the transmitter to effectively mitigate the degradation of the transmitted signal caused by atmospheric turbulence. At the same time, M-th quadrature amplitude modulation (M-QAM) signals have become a hot topic in recent research on new modulation methods due to their high-frequency band utilization and anti-noise performance. With this modulation method, the data transmission rate can reach tera bits per second (Tbit/s). In this context, turbulence changes in the atmospheric channel can be regarded as a slow fading process. Under these conditions, it is feasible to adaptively adjust the optimal probability distribution of the transmitted signal using genetic algorithms, based on the real-time acquired turbulent channel state information in combination with probabilistic shaping technology. Furthermore, by combining geometric shaping technology, the signal’s resilience to turbulence can be further enhanced, thereby improving communication quality. In this study, research is conducted under the Gamma-Gamma turbulent channel model. The proposed scheme integrates adaptive probabilistic shaping with geometric shaping based on channel conditions and provides a system model. This scheme can effectively improve the generalized mutual information of QAM signals and alleviate the effect of atmospheric turbulence on communication systems.

Based on the phase noise generated by the electrical demodulation module, treated as Gaussian noise, we consider the influence of different turbulence intensities on the laser signal. It obtains the turbulent channel state information in real time through the scintillation index calculation device and feeds this information back to the transmitter. An adaptive communication system model is then constructed based on the channel state. At the transmitter, the optimal probability distribution of the transmitted signal is determined through iterative optimization using a genetic algorithm, with the maximum generalized mutual information as the objective. The transmitted signal is adjusted to the optimal distribution through a distribution matcher. Next, combined with geometric shaping technology, the square QAM constellation is transformed into a symmetrically distributed circular arrangement, which realizes a research scheme that integrates adaptive probabilistic shaping with geometric shaping. For the low signal-to-noise ratio under different scintillation indices, we calculate and analyze the generalized mutual information of the proposed scheme, the bit error rate before forward error correction decoding, and the normalized generalized mutual information.

To solve the problems of high bit error rates and low generalized mutual information caused by the influence of atmospheric turbulence on uniformly distributed signals in atmospheric channels, we propose an adaptive probabilistic shaping technique combined with geometric shaping, which is based on time-varying turbulent channel state information under QAM modulation. Compared with uniform distribution or the application of single probabilistic shaping or geometric shaping, the constellation diagram obtained by the scheme proposed in this paper not only significantly reduces the aliasing phenomenon (Fig. 6) but also facilitates signal judgment. In terms of generalized mutual information, the signal after joint shaping shows a gain of 0.07 bit/symbol compared with uniform distribution (Fig. 7). In addition, when the scintillation index is 0.1, the bit error rate of the joint shaping scheme proposed in this paper is reduced from 7.6×10^-2 to 9.5×10^-3 (Fig. 9), which achieves an order-of-magnitude improvement compared with uniform distribution.

We propose a technical scheme for adaptive probabilistic shaping combined with geometric shaping based on channel conditions. At the transmitting end, the optimal probability distribution of the transmitted signal is explored using a genetic algorithm. The proposed scheme is simulated under turbulence intensities corresponding to different scintillation indices, and the generalized mutual information, normalized generalized mutual information, and bit error rate performances of the four schemes—uniform distribution, geometric shaping, adaptive probabilistic shaping, and adaptive probabilistic shaping combined with geometric shaping—are analyzed in detail. The results show that, compared with the uniform distribution or single constellation shaping schemes, the adaptive probabilistic shaping combined with the geometric shaping scheme proposed in this paper achieves the best communication performance. It can reduce the bit error rate of the communication system, improve the generalized mutual information of the system, and alleviate the influence of atmospheric turbulence on the signal to some extent. Due to experimental limitations, we only simulate and verify the proposed scheme. In the actual communication process, to ensure the smooth implementation of this scheme, the optimal distribution lookup table for the transmitted signal can be pre-established through simulation experiments to ensure that signal shaping and transmission are completed quickly within the atmospheric coherence time. In summary, the adaptive probabilistic shaping combined with the geometric shaping scheme proposed in this paper provides a new approach to improving the performance of laser communication systems.