Driven by the global data surge, free-space optical (FSO) communication at 1550 nm has become essential for 6G backhaul and inter-satellite links due to its anti-electromagnetic interference capabilities. However, atmospheric turbulence and beam misalignment reduce single-mode fiber efficiency and limit practical application. To address this, a hybrid architecture combining few-mode/multimode fiber (FMF/MMF) with FSO is proposed. Mode-division multiplexing (MDM) at the receiver decomposes signals into spatial modes, which are reconstructed using modular diversity reception and adaptive multiple input multiple output-digital signal processing (MIMO-DSP), effectively suppressing turbulence and misalignment. This system improves robustness through mode/polarization diversity, reducing inter-mode crosstalk via optimized DSP, and using cost-effective 1550 nm components. Despite challenges in receiver-side signal synthesis—such as hardware complexity in coherent schemes and high cost in programmable interrupt controller (PIC) solutions—a novel FMF-FSO structure is introduced. It integrates time-division multiplexing reception and a least mean squares/ constant modulus algorithm (MIMO-LMS/CMA)-based maximal ratio combining (MRC) algorithm. With six spatial modes and polarization multiplexing, 12-channel 32 GBaud quadrature phase shift keying (QPSK) transmission is achieved. We hope this architecture maintains resilience against hybrid channel damage, and its compatibility with the existing single-mode infrastructure can improve the application of adaptive fiber-FSO systems in future optical networks.

To address the hardware redundancy of multiple-receiver structures in traditional FMF-FSO systems, we propose a time-domain diversity (TDM)-based receiver architecture (Fig. 1) for comprising three modules: 1) A signal generator outputs a rectangular wave with a period T=80 μs, triggering an acousto-optic modulator (AOM) to produce a pulse width ΔT=8 μs. 2) A mode demultiplexer decomposes the input beam into six orthogonal modes, which are truncated by the AOM and injected with gradient delays (ΔL=2 km, total delay Δτ=6.66 μs) into standard single-mode fiber (SSMF). 3) An optical coupler (OC) incoherently combines these time-divided signals into a serial output. This setup allows multi-mode signal reception via a single channel, reducing hardware complexity by 83.3% compared to traditional multi-channel setups. The coherent receiver uses MIMO-LMS/CMA and MRC algorithms to suppress inter-channel crosstalk and enhance signal-to-noise ratio (SNR) without increasing bandwidth, maximizing spectral efficiency. In transmission, dual-polarized quadrature amplitude modulation (QAM) signals are generated from pseudo-random binary sequence (PRBS), shaped with a roll-off factor α=0.01, and modulated at 1550 nm using in-phase/quadrature (I/Q) modulators. Signals are polarization-multiplexed, amplified, and combined via a mode multiplexer into 12 channels over FMF. A tunable variable optical attenuator (VOA) simulates FSO turbulence and misalignment by axial fiber displacement (0?15 μm). After space-mode separation, the TDM receiver uses gradient-delayed SSMF to recombine signals, which are then coherently detected and processed digitally. Advanced DSP compensates for timing offsets and I/Q imbalance using generalized serial orthogonalization process (GSOP) and the joint MIMO-LMS/CMA-MRC algorithm, which mitigates polarization mode dispersion (PMD), mode coupling, differential group delay (DGD), and mode-dependent loss (MDL)—demonstrating the TDM receiver’s feasibility in FMF-FSO hybrid systems.

The bit error rate (BER) performance of a 6-mode 32 GBaud polarization division multiplexing (PDM)-QPSK system using the MIMO-LMS/CMA-MRC algorithm is evaluated across a received optical power range from -6 dBm to 8 dBm. The input power is measured at the mode demultiplexer, and the best BER among all 12 channels is selected as control group to reduce the influence of polarization and mode coupling. Experimental results show that the BER consistently stayed below the 7% hard-decision FEC threshold (3.8×10^-3), surpassing traditional DSP schemes which meet this threshold only when the received power reaches -2 dBm (Fig. 4). Even at -6 dBm, the system achieves a BER of 3.3×10^-3, demonstrating high sensitivity. At powers ≥2 dBm, near error-free transmission is achieved (BER<7.6×10^-6). Notably, the noise suppression gain of the algorithm increases with channel attenuation, with an average 4 dB improvement in sensitivity. Meanwhile, the constellation diagrams at 0 dBm further confirmed the effectiveness of the algorithm due to linear compensation and optimal combining. To evaluate the impact of modal delay, a tunable delay line (0?10 km) is introduced to decorrelate the six modes. In conventional DSP, BER exceeds the hard-decision FEC threshold due to modal delay. However, the proposed algorithm suppresses BER to 3.2×10^-3 at ≥-2 dBm. At 8 dBm, BER increased due to receiver saturation. Even with delay lines, the algorithm reduces signal error, validating its robustness (Fig. 5). A short-distance FSO experiment using a graded index (GRIN) lens and adaptive receiver further confirmed its effectiveness. Compared to a benchmark BER of 1.3×10^-3 in traditional FSO systems, the FMF-FSO system with the proposed algorithm achieves near error-free performance, showing three orders of magnitude improvement and strong resistance to atmospheric fading and spatial coherence degradation (Fig. 6).

We propose a novel few-mode fiber-free space optical (FMF-FSO) transmission system, integrating TDM receiver with improved MIMO-LMS/CMA-MRC equalization algorithm. The architecture employs PDM and MDM to simultaneously deliver 32 GBaud QPSK signals across 6 linearly polarized modes (LP 01, LP 02, LP 11a/b, LP 21a/b), establishing a 12-channel multiplexing framework. At the receiver DSP process, a hybrid MIMO-LMS/CMA-MRC equalization algorithm achieves an average improvement of 4 dB in power sensitivity by adaptively optimizing weight matrices, effectively mitigating atmospheric disturbances and alignment fluctuations in FSO transmission. Experimental validation demonstrates a BER of 3.3×10^-3 with received optical power of -6 dBm, compliant with 7% hard-decision FEC thresholds. The system’s compatibility with existing single-mode infrastructure and demonstrated resilience to hybrid channel impairments position it as a viable candidate for next-generation long-haul optical networks requiring adaptive fiber-FSO convergence.