Facing the exponential growth of global data traffic^[1], modern communication networks are in urgent need of breakthrough technologies to provide greater bandwidth and higher transmission rates. The terahertz (THz) band (0.1–10 THz) has become a core candidate technology for 6G communication systems due to its rich spectrum resources and high-speed transmission potential^[2]. However, traditional electronic devices have significant bandwidth limitations in generating and processing high-speed THz signals^[3]. This technical bottleneck has given rise to photon-assisted THz signal generation technology, which has successfully achieved ultra-wideband and large-capacity data transmission by fully leveraging the high-frequency response, wide-band characteristics, and low transmission loss advantages of optical devices^[4]. It is worth noting that the effective coverage range of current photon-assisted THz wireless systems is generally limited to tens of meters. When the transmission distance is extended to hundreds of meters, its transmission rate will be significantly attenuated. Therefore, breaking through the technical barriers of large-capacity and long-distance photon-assisted THz communication is of key significance to promoting the practical application of this technology^[5].

Using probability shaping (PS) technology to reduce the average transmission energy per bit to improve wireless coverage, Wang et al. achieved 1.41 m wireless transmission with a 25.6 Gbit/s PS-16QAM signal at 450 GHz^[6] and 26.8 m wireless transmission with single-channel 106.2 Gbit/s^[7] data rates at 350 GHz. By increasing the power budget of the THz link, Nagatsuma et al. achieved 100 m wireless transmission at 300 GHz with a single-channel 50 Gbit/s data rate using a pair of 54 dBi high-gain antennas^[8]. Harter et al. achieved 58 m wireless transmission at 310 GHz with single-channel 10 Gbit/s data rates using a THz power amplifier and a THz low noise amplifier^[9]. Castro et al. achieved THz wireless transmission of 32 GBaud quadrature phase-shift keying (QPSK) and 16QAM signals at 300 GHz with a broadband positive-intrinsic-negative (PIN) photodiode^[10]. Ummethala et al. used a plasmonic-optic hybrid (POH) modulator as the front end of a THz receiver to achieve 16 m wireless transmission with a 50 Gbit/s QPDK signal at 288.5 GHz^[11]. Jia et al. used a monolithic integrated dual distributed feedback (DFB) laser to achieve a 16-QAM orthogonal frequency-division multiplexing (OFDM) signal 10.7 m wireless transmission at 408 GHz with a single channel and a speed of 131.21 Gbit/s^[12]. Li’s team used high-gain lenses and horn antennas (HAs) to achieve 850 m wireless transmission with 50 Gbit/s data rates at 320 GHz^[13].

The above-mentioned THz communication systems still have two significant technical bottlenecks. First, the existing systems often need to sacrifice the transmission rate when extending the wireless transmission distance, while the ideal goal should be to achieve a synergistic improvement of the two. Second, single polarization links are generally used for long-distance transmission verification. In order to optimize the rate-distance product (AIUR), it is necessary to integrate multi-dimensional multiplexing technology, high-gain THz devices, and innovative applications of digital signal processing (DSP) algorithms. Ding et al. used a pair of Teflon lenses and PS technology to achieve 104 m transmission of 339 GHz 16 GBaud PS-256QAM signals with 124.8 Gbit/s^[14]. Zhu et al. established a single-channel 100 Gb/s and dual-channel 200 Gb/s system in the 360–430 GHz band^[15]. The nonlinear impairments introduced by the fiber transmission, photoelectric detection, and amplification links are particularly significant in long-distance systems that require high transmission power. Algorithms such as the multiple-input multiple-output (MIMO) structure Volterra nonlinear equalizer (VNE) algorithm have shown significant effects in reducing the bit error rate (BER), but their computational complexity increases sharply with the high-order memory length^[16]. The MIMO hybrid time-frequency polarization domain triangular memory polynomial nonlinear equalizer (MIMO-TFPD-TMP-NLE) can significantly reduce the amount of multiplication operations by performing trigonometric operations on the received signal^[17].

However, under high-power conditions, the saturation nonlinear effects of photodiodes and the low-noise amplifier (LNA) are significant, and the compensation capability of the above-mentioned non-machine learning algorithm cannot meet the requirements. In fiber radio (RoF) systems, machine learning (ML) is widely used to compensate for the nonlinear effects of optoelectronic devices^[18,19]. ML techniques learn system impairments from training data, enabling simultaneous mitigation of multiple nonlinear effects. Liu et al. demonstrated that deep neural networks (DNNs) outperform Volterra equalizers in RoF systems, and Zhou et al. further reduced BER using complex-valued neural networks (CVNNs)^[20,21]. Wang et al. achieved sub hard-decision forward error correction (HD-FEC) threshold transmission in a dual-polarization 30 GBaud 320 GHz THz link using two-dimensional convolutional neural networks (2D CNNs)^[22]. Recent studies also employ recurrent neural networks (RNNs) and long short-term memory (LSTM) networks for signal equalization^[23], while iterative pruning enables neural network compression without performance loss^[24].

Existing long-distance THz communication systems mainly use a single-polarization architecture. Due to the limited freedom of the polarization dimension, there is a fundamental limitation on the transmission capacity. In order to break through this bottleneck, we adopted a 2×2 MIMO system that combines polarization multiplexing and spatial multiplexing technology. Through polarization multiplexing and spatial multiplexing technology, multi-channel parallel transmission can be achieved, the data rate can be increased exponentially, the system capacity can be improved, and the high-speed and large-capacity requirements of 6G communication can be met. In addition, 2D CNN technology based on iterative pruning is adopted, which effectively compensates for nonlinearity while maintaining a lower computational complexity compared to existing neural network algorithms.

In this paper, we demonstrate a large-capacity 200 Gb/s photonics-assisted THz wireless system over a 200 m outdoor 2×2 MIMO link in the 300 GHz band, resolving the rate-distance trade-off problem through three key innovations. First, a polarization-enhanced MIMO architecture is adopted. The co-polarized 2×2 MIMO configuration provides intrinsic diversity gain while maintaining compatibility with optical polarization-division multiplexed (PDM) networks, eliminating active polarization alignment. Second, an adaptive 2D CNN equalizer with iterative pruning is employed. This equalizer jointly compensates polarization crosstalk and temporal nonlinearities, reducing the BER by up to an order of magnitude at 16 dBm compared to conventional DSP. Iterative weight pruning achieves 80% sparsity, with 59.7 × 10³ multiply-accumulate operations (MACC), 30.2% lower than 1D CNN. Finally, the pruned model meets the HD-FEC threshold at 6 dBm, making it suitable for low-power THz devices.

In our system, we transmit 50 Gbaud polarization-division multiplexed quadrature phase-shift keying (PDM-QPSK) signals, which is the highest reported rate for 200 m THz links, to the best of our knowledge. This represents a crucial step toward practical long-distance THz systems, providing a scalable blueprint for 6G infrastructure.

In RoF systems, signals may be affected by multipath effects, chromatic dispersion (CD), and frequency synchronization errors. As a result, the received millimeter wave (MMW) signals simultaneously suffer from inter-symbol interference (ISI) and inter-carrier interference (ICI) distortions. Traditional polarization demultiplexing techniques have limited effectiveness in equalizing these distortions. Given the exceptional nonlinear fitting capabilities of neural networks, we employed both 1D CNN and 2D CNN in our experiments to recover the signals.

In 1D CNN, real and imaginary signal components are processed separately. As illustrated in Fig. 1(a), it has convolutional, batch normalization, activation, pooling, and fully connected layers. The principle of 1D convolution is illustrated in Fig. 1(b). 1D convolution slides a kernel over 1D data. In 2D CNN, real and imaginary parts are input together. As depicted in Fig. 1(c), it has similar layers but with 2D operations. The principle of 2D convolution is shown in Fig. 1(d). The convolution kernel slides over the 2D input data in both horizontal and vertical directions.

In terms of optimizer selection, the Adam optimizer combines the advantages of momentum and root mean square propagation (RMSProp), and its adaptive learning rate can track channel changes faster than stochastic gradient descent (SGD) with a fixed learning rate or traditional momentum methods. In addition, in a pruned 2D CNN, a large number of weights are reset to zero. Adam avoids invalid updates of zero gradient parameters through bias correction and maintains sparsity. Therefore, these CNN architectures use Adam optimizer to minimize mean-square error (MSE) and, thus, improve accuracy.

Applying CNN to THz channel equalization overcomes traditional algorithm limits, compensating for channel distortions and improving transmission. By applying the CNN algorithm, we are able to fully leverage CNN’s exceptional capabilities in feature extraction and pattern recognition. This enables us to overcome the limitations of traditional algorithms when dealing with complex channel conditions, compensating for channel distortions and improving transmission.

Although neural networks are powerful, they consume significant storage, memory bandwidths, and computational resources. Research shows that neural networks contain a large number of redundant neurons and weights, with only 5%–10% of the total weights participating in the main calculations and influencing the final results. Neural network pruning is an optimization technique that reduces network complexity by removing unimportant weights or neurons, thereby improving runtime efficiency, reducing storage requirements, and potentially enhancing generalization capabilities. The key pruning steps include: training the complete network until it performs well, defining pruning criteria, ranking and removing unimportant weights or neurons based on the criteria as shown in Fig. 2, and finally retraining the network to fine-tune the remaining weights and restore performance. In data processing, we employ threshold-based pruning using standard deviation (STD) as the metric, setting weights below the threshold to zero. Iterative pruning allows the network to gradually adapt to pruning changes, avoiding the performance degradation caused by one-time pruning and ensuring normal network functionality after each pruning step.

Figure 3 shows the detailed DSP flow at the receiving end. After baseband down-conversion, resampling, and in-phase and quadrature-phase (IQ) imbalance compensation, an adaptive blind polarization demultiplexer based on the T/2 constant modulus algorithm (T/2 CMA) with least mean square (LMS) iteration is used for equalization. Subsequently, Viterbi frequency offset estimation (FOE) and blind phase search (BPS) carrier phase recovery (CPR) are carried out to eliminate the frequency offset and laser phase noise generated by two free-running external cavity lasers (ECLs). It should be noted that such hybrid fiber-THz wireless links have non-negligible linear and nonlinear impairments, mainly originating from the dispersion and Kerr nonlinearity of the fiber channel, the fading effect of the wireless channel, the nonlinearity of the modulator, and the saturation distortion of the electrical/optical amplifier. These linear and nonlinear impairments ultimately manifest as linear and nonlinear crosstalk between adjacent symbols in the transmission sequence, which can be overcome by the corresponding linear and nonlinear equalizers.

In this experiment, three different neural network equalizers are adopted and compared, including the fully connected neural network (FCNN), 1D CNN, and 2D CNN. The schemes using the FCNN and 1D CNN employ two independent neural networks to compensate for the linear impairments of the real-part and imaginary-part signals, respectively. The scheme using the 2D CNN inputs the real part and the imaginary part into the neural network simultaneously for joint equalization.

Figure 4(a) shows the photonics-assisted experimental setup for 200 Gb/s signal transmission over a 200 m outdoor 2×2 MIMO THz wireless link. At the optical transmitter, a QPSK baseband signal with a roll-off factor of 0.01 is generated offline by MATLAB and loaded into a 92 GSa/s arbitrary waveform generator (AWG) to generate the QPSK signal. These signals are then amplified by two parallel electrical amplifiers (EAs) with a gain of 25 dB and modulated using an I/Q modulator with a 3 dB bandwidth of 22 GHz. An ECL operating at 193.5 THz with a wavelength of 1549.316 nm serves as the optical carrier, and its optical power is 14 dBm. The modulated signal is amplified by an erbium-doped fiber amplifier (EDFA), transmitted through a 20 m standard single-mode fiber (SSMF), and then coupled with a 10 dBm local oscillator (LO) light source emitted by a laser operating at 193.2 THz with a wavelength of 1551.720 nm via an optical coupler (OC). The linewidth of both ECLs is 100 kHz. Two polarization controllers (PCs) are used to control the polarization states. A polarization beam splitter (PBS) is used to separate the X and Y polarization signals and input them into the uni-traveling carrier photodiodes (UTC-PDs). The frequency interval between the two light waves is 300 GHz, generating a THz signal with a carrier frequency of 300 GHz. To achieve polarization-diversity optical heterodyne detection, a PBS is used to split the signal into X and Y polarization signals with equal power. Subsequently, these two orthogonal polarization signals are converted into 300 GHz horizontally (H) and vertically (V) polarized THz signals through horizontally and vertically placed UTC-PDs, respectively. After being amplified by THz LNAs with a gain of 22 dB, the H- and V-polarized THz signals are simultaneously transmitted through 48 dBi high-gain cylindrical lens horn antennas (CLHAs) for 200 m 2×2 MIMO wireless transmission.

At the THz wireless receiver, the divergent THz signals in each polarization direction are first focused by a self-made polytetrafluoroethylene (PTFE) lens with a diameter of 30 cm and a focal length of 50 cm, and then input into a THz HA with a directivity gain of 26 dBi. After that, the THz signals received in each polarization direction are amplified by an LNA to compensate for the power loss. Then, frequency down-conversion is carried out through an integrated harmonic mixer with a multiplication factor of 24. A radio frequency (RF) signal with an output power of 9 dBm is evenly distributed by a 40 GHz power divider and then connected to the LO ports of the two THz harmonic mixers, respectively. After that, the two obtained intermediate-frequency (IF) signals are amplified by EAs with a gain of 35 dB and captured by a real-time 80 GSa/s digital storage oscilloscope (OSC). Finally, the obtained PDM-QPSK signals are further demodulated at the receiving end by offline digital coherent DSP (see Fig. 3 for details). Figures 4(b)–4(e) show the experimental photos of the THz transmitter and receiver.

This experiment was conducted in sunny weather, with the temperature ranging from 23°C to 28°C and the relative humidity being 40%. The map measurement results of the 200 m wireless transmission link in the experiment are shown in Fig. 5. To accurately align the transmitter and receiver of the photonics-assisted THz long-distance transmission system, while using tripods and telescopes to calibrate the positions, we also transmit a single-frequency signal from the transmitter and observe the power of the received signal through an OSC. The calibration is completed when the power reaches its maximum value.

For this experiment, the wireless link power budget of the communication system was computed, as characterized by the Friis formula, PR=PT+GT+GR−20log(4πdfc)−Lm,(1)where PT represents the transmit power, GT denotes the gain of the transmitting antenna, GR is the gain of the receiving antenna, and d, f, and c represent the wireless transmission distance, the frequency of millimeter-wave signals, and the speed of light in vacuum, respectively, while Lm denotes the atmospheric loss.

In our experiment, the output power (PT) is −3.5dBm. The gain of CLHA (GT) is about 48.5 dBi, and the combined gain of the lens and HA (GR) is about 66 dBi. Because the wireless transmission distance (d) is 200 m and the frequency of the millimeter-wave signal (f) is 300 GHz, the free space path loss 20log(4πdfc) equals approximately 128 dB. On a sunny day, the atmospheric loss (Lm) is about 1 dB at 300 GHz for a 200 m wireless distance link.

Figure 6 shows the constellation diagrams at three different powers of 0, 6, and 16 dBm in X-pol, showing the performance comparison under different signal processing methods. The changes in the color and density of the constellation diagram reflect the strength and distribution of the signal. It can be seen that, as the power increases, the points in the constellation diagram are more concentrated, and the BER decreases.

When the input optical power is 16 dBm, the BER of the pruned 2D CNN is slightly higher than that before pruning. This is because, as the transmit power increases, the power of the input PD increases, and the nonlinear distortion caused by the PD saturation effect worsens. At this time, iterative pruning has limited ability to further improve the BER performance. In order to minimize the model complexity while keeping the BER performance almost unchanged, we adopted an algorithm based on iterative pruning. Although pruning sacrifices the BER performance of 1×10−4, the computational complexity is reduced by 72% compared to the unpruned model, which means faster processing speed and lower energy consumption, which is crucial for high-speed communication systems.

Figure 7 shows that the performance of the neural network-based equalizer is significantly better than the traditional DSP method when transmitting a 50 Gbaud PDM-QPSK signal over a 200 m 300 GHz wireless link, especially in the high-power region of 6–16 dBm. Among the neural network methods, the 2D CNN shows the best BER performance because the 2D convolution kernel effectively compensates for the UTC-PD nonlinearity and channel impairments. Taking X-pol as an example, the 2D CNN reduces the BER to 1.9×10−3 at 16 dBm, an order of magnitude higher than 1.9×10−2 before the neural network. In contrast, the BER of the 1D CNN at 16 dBm is 2.9×10−3, which is higher than that of the 2D CNN. This is because it only processes the time dimension and cannot capture the crosstalk between polarizations. The FCNN has a lower BER performance of 7.2×10−3 at 16 dBm due to the global parameter coupling of the fully connected structure and limited nonlinear compensation ability.

Furthermore, the network pruning strategy iteratively removes redundant weights. The pruned model improves robustness while retaining key feature extraction capabilities, further optimizing performance. For example, the BER of the pruned 2D CNN reaches 3.8×10−3 at 6 dBm, which meets the HD-FEC threshold and verifies its real-time feasibility in THz systems. The experiment did not show BER deterioration caused by pruning, indicating that the strategy effectively retains the core parameters. In addition, it can be seen from Fig. 7 that in the low-power area less than 6 dBm, the signal is dominated by thermal noise. As the input optical power increases from 0 to 4 dBm, the BER of the pruned 2D CNN decreases from 1.74×10−2 to 1.14×10−2, a decrease of 34.5%. At the threshold inflection point of 4–6 dBm, the SNR exceeds the critical value, the 2D CNN accelerates the compensation of the PD saturation effect through deep nonlinear mapping, the slope of the BER curve increases significantly, and the BER drops sharply. When the input optical power increases to 6 dBm, the BER of the pruned 2D CNN drops to 3.2×10−3, which is a 71.9% decrease compared to the input optical power of 4 dBm. In the high-power region greater than 6 dBm, the 2D CNN uses joint feature extraction to fully compensate for the PD saturation effect, so that the BER is stabilized below the HD-FEC threshold.

Figure 8 shows that the network sparsity of all models increases monotonically with the increase of input optical power. This is because the SNR of high-power signals increases, which enables the pruning algorithm to more safely remove redundant parameters without significantly affecting the BER. 2D CNN always achieves the highest sparsity under the same polarization. Its advantage comes from the fact that the 2D convolution kernel can extract more redundant features that can be pruned, while the fully connected structure of the FCNN has limited pruning space due to the tight coupling of parameters. It is worth noting that the sparsity of Y-pol signals is generally slightly higher than that of X-pol (e.g.,  2D CNN sparsity Y-pol: 68% versus X-pol: 66% at 12 dBm), which may be related to the need for more conservative pruning due to the dominant system nonlinearity (such as UTC-PD response) of X-pol signals, although Y-pol has a slightly higher BER due to environmental interference (such as polarization mode dispersion). In the 4–8 dBm threshold range, the sparsity growth rate increases significantly, and the BER curve simultaneously shows an inflection point, indicating that the pruning algorithm can efficiently identify non-critical parameters under high SNR. When the power exceeds 8 dBm, the sparsity growth slows down, indicating that the model approaches the extreme sparsity of the architecture, and a trade-off between the compression rate and performance is needed to avoid destroying the core feature channels. These phenomena provide an important basis for the design of dynamic pruning strategies and polarization adaptive compression algorithms in THz systems.

Figure 9 shows the maximum sparsity data when the HD-FEC threshold is met in the power range of 10–16 dBm. The sparsity of all models shows an increasing trend in this range, but the growth rate gradually slows down with the increase of power. Among them, the sparsity of 1D CNN increases slightly, approaching the architectural limit. The sparsity of 2D CNN increases by up to 6%, indicating that it still maintains compression potential. 2D CNN can achieve up to 80% sparsity while meeting the HD-FEC threshold, which is significantly better than 1D CNN, thanks to the inherent redundancy of its joint feature extraction. The polarization-related sparsity difference reveals the asymmetry of channel impairments and provides a basis for differentiated pruning of MIMO systems.

Figure 10 shows the MACC complexity of different equalizers in the Y-pol. It shows that pruning can significantly reduce the complexity of neural networks, with 2D CNN benefiting the most. With only 11% higher complexity than FCNN-pruned [(59.7×103 − 53.8×103)/(53.8×103) = 11%], 2D CNN reduces BER by 3.8 times, from 7.2×10−3 to 1.9×10−3, proving that it retains stronger feature extraction capabilities and efficiently utilizes computing resources. Compared with the pruned 1D CNN, the complexity of the 2D CNN is reduced by 30.2% [(85.5×103 − 59.7×103)/(85.5×103) = 30.2%]. Under the condition of a high signal-to-noise ratio of 16 dBm, 2D CNN-pruned achieves the best balance between computational complexity and BER performance.

In this paper, we investigate the optimization of deep-learning-based equalizer architectures and model compression strategies for 50 GBaud PDM-QPSK signal transmission over a 200 m wireless link in the 300 GHz THz band. The experimental results demonstrate that pruned 2D CNN effectively balances model complexity and performance in THz MIMO systems, achieving both computational efficiency and robustness. By leveraging 2D convolutional kernels to adaptively capture polarization crosstalk and temporal nonlinearities in MIMO channels, the proposed architecture reduces the BER to 1.9×10−3 at 16 dBm, outperforming conventional DSP methods by an order of magnitude. The model after pruning exhibits 77% sparsity, and its MACC complexity is significantly lower than that of 1D CNN, validating its potential for real-time deployment. Notably, the pruned model satisfies the HD-FEC threshold at 6 dBm, providing a feasible equalization solution for low-power THz devices. This approach, particularly when integrated with photonics-assisted signal generation and polarization-diversity reception, offers valuable insights for future THz real-time communication systems.