Optical spatial modulation is a novel multiple input multiple output (MIMO) technology that activates a single transmitting antenna at each moment to avoid co-channel interference between channels. However, it has low spectrum utilization and significant limitations. Optical generalized spatial modulation (OGSM) extends this approach by enabling multiple antennas to transmit data simultaneously, improving antenna utilization and overall data transmission rate. In OGSM systems, bit error performance can be enhanced by refining detection algorithms. However, these improvements have often been limited. Therefore, researchers have turned to antenna selection and power allocation algorithms to optimize bit error performance more effectively.

In this paper, we propose a norm joint similarity coefficient-based antenna selection algorithm and a particle swarm optimization-based power allocation algorithm. To maximize channel capacity, we derive a mathematical model that reflects the influence of channel norm and similarity coefficient on capacity. The norm joint similarity coefficient is used to create an antenna selection strategy that optimally combines antennas for improved performance. A channel capacity-based fitness function is designed using the particle swarm optimization algorithm to allocate optimal power to selected antennas, thus enhancing system transmission quality.

In the simulated environment, we apply BPSK modulation with two active antennas. The following key results emerge from the analysis: 1) at low signal-to-noise ratios (SNRs), theoretical BER of the OGSM-MIMO system is initially higher than simulated bit error rate (BER); however, as SNR increases, this gap narrows, aligning closely at higher SNRs. 2) As the number of receiving antennas increases, bit error performance improves notably. For instance, when BER reaches 10^-3, the four-antenna setup outperforms the three-antenna setup by 4.1 dB (Fig. 2). The proposed norm joint similarity coefficient antenna selection algorithm significantly enhances bit error performance compared to random, RSS, and norm-based selection algorithms. When BER reaches 10^-4 with four transmitting antennas, bit error performance improves by 7.9 dB, 5.2 dB, and 2.2 dB, respectively. With six antennas, improvements are 7.4 dB, 5.8 dB, and 3.2 dB, respectively (Fig. 3). In addition, particle swarm optimization-based power allocation algorithm considerably enhances bit error performance over traditional equal-power and water-filling methods, improving by 7.5 dB and 4.2 dB, respectively, at BER of 10^-3 (Fig. 4). In the antenna selection algorithm system utilizing the norm joint similarity coefficient, when BER reaches 10^-4, the bit error performance of the OGSM6×4-4 system is enhanced by 4.5 dB following particle swarm optimization-based power allocation, while the OGSM4×4-2 system achieves a 2.6 dB improvement under the same optimization. Compared to the OGSM4×4-2 system, the OGSM6×4-4 system exhibits a 3 bit/s increase in each control unit, though its bit error performance slightly declines (Fig. 5).

In this paper, we examine the norm joint similarity coefficient antenna selection and particle swarm optimization power allocation algorithms for the visible OGSM-MIMO system, providing a simulation-based analysis of system bit error performance. The findings indicate that the norm joint similarity coefficient algorithm plays a critical role in the system’s operation, enabling intelligent antenna activation that mitigates co-channel interference and improves system capacity and stability. The simulation results confirm that, across different SNR conditions, BER is significantly enhanced with this algorithm over traditional methods. In addition, the particle swarm optimization-based power allocation strategy optimizes transmission power, allowing the system to adapt to varying communication environments and channel conditions, thus improving transmission efficiency and performance. Overall, the system employing the particle swarm optimization algorithm achieves a lower BER across diverse channel conditions compared to conventional methods.