In this paper, the objective of the research is to improve the performance of Orthogonal Frequency Division Multiplexing (OFDM)-Wavelength Division Multiplexing (WDM) optical fiber communication system. a Quadrature Grouped Subcarrier Amplitude Shaping (QGSAS) dual-Phase Conjugate Twin Waves (dual-PCTW) scheme is proposed, which can suppress the nonlinear effect of the systems. In addition, the simulation systems under different conditions are built to explore the performance of the scheme. The specific experimental methods are as follows.

With the mixed programming of Optisystem and Matlab, a QGSAS dual-PCTW OFDM system is simulated based on 16-ary and 64-ary Quadrature Amplitude Modulation (QAM) formats. Two kinds of QGSAS (sinusoidal amplitude shaping, rectangular amplitude shaping) and three dimensions dual-PCTW (time domain, polarization domain, sub-carrier domain) are adopted for the system to form 12 different combinations.

The simulation results show that the QGSAS dual-PCTW scheme can effectively suppress the nonlinear effects in the system. In several combination schemes discussed, the 16QAM dual-PCTW scheme in time domain has the best performance. When the QGSAS technology is used, the scheme of rectangular amplitude shaping is better than the sinusoidal amplitude shaping.

Therefore, it can be concluded that in the scheme studied in this paper, the rectangular amplitude shaping scheme combined with the time domain scheme dual-PCTW technology under 16QAM can effectively suppress the nonlinear effect of OFDM-WDM system.