Optical orthogonal frequency division multiplexing (OOFDM) technology, widely employed in free-space optical (FSO) communication, faces challenges such as slow out-of-band attenuation and limited spectral efficiency. These issues are especially pronounced in environments affected by atmospheric turbulence, which further reduces the effectiveness of OOFDM. To address these challenges, we propose a novel transmission scheme that combines faster than Nyquist (FTN) signaling with optical filter bank multicarrier (OFBMC) technology, creating the OFBMC-FTN system aimed at enhancing spectral efficiency without sacrificing performance. Additionally, we introduce FBMCFormer, a Transformer-based detection algorithm that leverages the multi-head self-attention mechanism of the Transformer to improve detection performance, particularly in turbulent environments commonly encountered in FSO systems. This approach responds to the growing demand for higher data transmission rates and more reliable communication systems, especially under challenging atmospheric channels.

Our study employs a dual approach, combining theoretical derivations and Monte Carlo simulations, to evaluate the proposed OFBMC-FTN system. We derive the theoretical bit error rate (BER) expression for the system using the maximum likelihood criterion and the Gamma-Gamma turbulence channel model, suitable for simulating atmospheric turbulence effects on optical signals in FSO systems. We assess system performance under varying turbulence intensities—weak, moderate, and strong—to ensure a comprehensive analysis across different conditions. A key innovation of this research is the FBMCFormer detection algorithm designed specifically for the OFBMC-FTN system. Utilizing Transformer networks, FBMCFormer captures both temporal and spatial dependencies in signal sequences. The multi-head self-attention mechanism within FBMCFormer prioritizes the most relevant signal components, thereby enhancing detection accuracy. FBMCFormer’s architecture consists of an input layer, a Transformer encoder, a one-dimensional convolutional layer, and a hard decision module (Fig. 4). While the convolutional layer extracts critical features from filtered signals, the Transformer captures long-term dependencies in turbulent environments. To validate the performance under different signal-to-noise ratios (SNRs) and turbulence intensities, Monte Carlo simulations were conducted. These simulations assess the BER performance of the OFBMC-FTN system and compare it with traditional modulation techniques such as direct current-biased optical filter bank multicarrier (DCO-FBMC), asymmetrically clipped optical filter bank multicarrier (ACO-FBMC), and OOFDM.

The OFBMC-FTN system achieves notable improvements in spectral efficiency compared to conventional optical multicarrier systems. With a packing factor of 0.9, the system improves spectral efficiency by 11.1% over DCO-FBMC, 94% over ACO-FBMC, and 120% over OOFDM systems (Fig. 3). These gains are achieved without increasing modulation order, as the FTN approach compresses symbol transmission intervals, enabling more efficient bandwidth utilization. For BER performance, our study shows favorable results across various turbulence intensities. As SNR increases, the simulation and theoretical BER curves converge, validating the theoretical model’s accuracy (Fig. 2). Under moderate turbulence, the system experiences a 2 dB loss in SNR compared to weak turbulence at the same BER level. FBMCFormer effectively addresses turbulence-related challenges. Compared to the maximum likelihood (ML) detection algorithm, FBMCFormer achieves near-optimal BER performance with significantly reduced computational overhead. Traditional deep neural network (DNN) detection algorithms struggle under varying turbulences due to rapid phase and amplitude fluctuations. In contrast, FBMCFormer adapts well to turbulent signal characteristics due to its multi-head self-attention mechanism, which captures both short- and long-term dependencies in the signal sequence, thus sustaining robust performance (Fig. 6). FFBMCFormer consistently outperforms DNNs while maintaining near-optimal BER performance compared to ML detection across various compression factors (Fig. 7). In terms of computational efficiency, FBMCFormer scales more effectively than ML detection (Fig. 8). While ML detection’s computational complexity increases exponentially with transmission frame numbers, FBMCFormer maintains relatively low complexity, making it a practical solution for processing large-scale signal sequences without compromising accuracy.

Our study demonstrates that integrating FTN signaling with OFBMC technology, along with Transformer-based detection algorithms, significantly enhances both spectral efficiency and signal detection accuracy in optical wireless communication systems. The proposed OFBMC-FTN system effectively addresses OOFDM limitations by reducing out-of-band leakage and improving spectral efficiency without additional bandwidth or higher modulation orders. Building on these enhancements, FBMCFormer reduces complexity compared to ML detection while maintaining near-optimal BER performance, particularly in turbulence-affected environments. Monte Carlo simulations validate the theoretical BER analysis, showing that the proposed system achieves superior spectral efficiency compared to existing technologies like DCO-FBMC, ACO-FBMC, and OOFDM. This study provides a promising direction for high-speed optical wireless communications with further potential for real-world applications. Future work will focus on refining system parameters and conducting real-world tests to further validate and optimize performance.