Vortex beams, distinguished by their unique spiral wavefront structure, phase singularity, and orbital angular momentum, offer possibilities for enhancing system performance. Notably, vortex beams with different modes are spatially orthogonal, enabling their use in orbital angular momentum multiplexing for augmenting the channel capacity and spectral efficiency. Moreover, demultiplexing these beams at the receiving end provides an additional boost to system capabilities.

This paper focuses on the application of coherent detection technology to a bi-directional slant path optical wireless communication system, employing orbital angular momentum multiplexing. This approach eliminates the need for image recognition steps, such as diffraction interference, thereby reducing errors introduced during the process. It analyzes the impact of the topological charge, altitude, and transmission distance on the performance of a vortex optical multiplex communication system. By incorporating an adaptive optics system for uplink and downlink correction, we can minimize crosstalk between modes, leading to improved detection sensitivity and channel capacity.

The architecture of the orbital angular momentum multiplexing coherent detection system for optical wireless communication using bi-directional slant transmission is illustrated in Fig. 1. Fig.2 presents a schematic view of the system's transmitting and receiving ends. At the transmitting end, the source signal laser is split into four beams using a 1×4 coupler. The signal for each channel undergoes external modulation after series-to-parallel conversion and is then transformed into a vortex beam, with topological charges of 1, 2, 3, and 4, through spiral phase plates. These channels are then combined with a 4×1 coupler for coaxial transmission. At the receiving end, a 1×4 coupler divides the beam into four channels, each mixed with local oscillator vortex beams with corresponding topological charges. Following balanced detection, the electric signal is recovered, demodulated, and then converted back into signals through parallel-to-series conversion, enabling signal transmission from the source to the end. The uplink and downlink employ a single adaptive optics system, located near the downlink receiver, for correction. This system's working principle is depicted in Fig. 3. Given the reversibility of the transmission link and the reverse superposition of the wavefront, the distortion of the uplink signal wavefront at the transmitting antenna is conjugate with the wavefront distortion of the downlink received by the receiving antenna. This feature allows for post-correction of the downlink and pre-correction of the uplink.

Fig. 4 illustrates the coherent gains of signal and local oscillator light with varying topological charges after mixing. These gains include uplink, downlink, corrected, and uncorrected scenarios. When the topological charge of the signal light (l_s1) is 1, and the topological charges of the local oscillator (LO) light (l_lo) are 1 and 2, the corresponding coherent gains stand at 0.867 and 0.156, respectively. Atmospheric turbulence, which is most potent near the surface, induces wavefront distortion in vortex light, reducing coherence between signal and LO light, and affecting their orthogonality. As a result, the downlink's correction effect is superior to that of the uplink. Fig. 8 reveals the bit error rate of each channel and the system's bit error rate under varying transmission distances. With increasing transmission distances, the wavefront distortion caused by strong turbulence in the uplink exceeds the adaptive optics' correction capability. Insets in Fig. 8 show the uncorrected and corrected light intensity and phase distribution after uplink and downlink transmission. The uplink utilizes pre-correction processing, leading to a larger corrected spot diameter compared with the uncorrected one. Wavefront correction does not influence the light intensity distribution, so no differences are observed in the light intensity distribution before and after downlink correction.

The study concludes that atmospheric turbulence can trigger mode crosstalk during vortex optical multiplexing transmission, and extending the transmission distance heightens the system's bit error rate. At the same transmission distance, mode crosstalk becomes more pronounced as the topological load increases. Adaptive optics is typically apt for phase compensation in weak turbulence conditions, with the correction effect of the downlink more obvious than that of the uplink. The vortex beam orbital angular momentum multiplexing coherent detection significantly enhances the system's detection sensitivity and channel capacity. These findings apply to coherent detection communication involving multiple orbital angular momentum multiplexing with an expanded multiplexing interval.