Since the establishment of the information theory in 1948, most researchers have focused on narrowing the gap with Shannon-Hartley theorem. The traditional rectangular quadrature amplitude modulation (QAM) is widely used in optical communication. Although this modulation scheme is relatively mature, the rectangular modulation format is still far from reaching the Shannon-Hartley theorem. To bridge the difference between rectangular QAM and Shannon-Hartley theorem, researchers have developed constellation shaping techniques, namely geometric shaping (GS) and probabilistic shaping (PS). These techniques are based on power constraints and designed around conventional points, such as 16QAM, 32QAM, and 64QAM. In the case of a Gaussian channel environment, the probability distribution scheme of PS is based on Maxwell-Boltzmann distribution. In this case, it is combined with GS to form geometric PS; however, the geometric PS of conventional points corresponds to its appropriate transmission rate. For example, the 16QAM's geometric probability shaping is suitable for transmitting signals with an entropy of about 3, but it causes performance issues when it is below 3. Additionally, it does not offer any advantage in PS when the entropy of the transmitted signal is above 3. Thus, this article aims to study the geometric PS scheme of unconventional and continuous points. This scheme can flexibly adapt to the channel environment and transmit appropriate information entropy.

It is necessary to focus on the PS scheme for GS to design a geometric PS scheme under power constraints. The probability distribution can be obtained from the Maxwell–Boltzmann distribution. This article designs the most compact hexagonal layout scheme in a two-dimensional plane. The distribution of noise in Gaussian channels is uniform in all directions, and thus, the constellation points are considered circles that conform to the noise distribution in Gaussian channels. After selecting a compact layout scheme, power screening is carried out. In power-limited schemes, layout selection is carried out to maximize space utilization, and points with low power are selected for modulation. Matlab Gaussian noise function is used to simulate the noise in the channel; linear regions in the experimental equipment are used for the experiments. The experiments focus on verifying the relationship between entropy and constellation points, while the selection of optical wavelength, signal rate, and power is secondary. The receiver in the experiment adopts a machine learning approach that can greatly reduce the complexity of the reception aspect. Moreover, machine learning intersects with traditional hard decision methods and has almost the same error rate in Gaussian channels.

This paper verified the coherent optical communication system with an information entropy of 3 and constellation points of 8-13, information entropy of 4 and constellation points of 16-23. The results show that, when the bit error rate is 5×10^-3, the geometric PS under hexagonal arrangement has gains of about 1 dB and 1.3 dB compared with 8QAM and 16QAM, respectively. Additionally, the simulation and experimental verification of geometric shaping at 8 and 16 points show a performance improvement of about 0.1 dB and 0.22 dB, respectively, compared with rectangular QAM. The essence of constellation shaping is to exchange complexity for performance improvement. Before the advent of machine learning, the complexity improvement in reception was not proportional to the benefits and was thus not widely used. However, this article adopts machine learning methods for signal reception, and the curve results also meet the expectations.

The geometric PS scheme under the power limitation proposed in this article was validated via simulation and experiments. Our findings show that the proposed scheme can achieve better bit error rates under the same power and signal-to-noise ratio conditions as the traditional scheme. However, the shaping scheme slightly increases the complexity of the system and results in varying signal-to-noise ratio gains under different signal-to-noise ratio conditions. Note that this article shows only representative cases, and the results show that at least 1 dB of gain can be obtained from the perspective of bit error rate. Moreover, as number of constellation points increases, the benefits obtained from the perspective of bit error rate also increase. In the experimental part, machine learning is applied to constellation reception decisions. Consequently, the cost of constellation shaping is gradually becoming acceptable. As machine learning technology becomes more mature, there will be opportunities to apply it to constellation shaping in channel environments other than Gaussian channels.