In order to explore the mechanism of IQ imbalance, combining polarization multiplexing(PDM), hexadecimal orthogonal amplitude modulation(QAM), phase conjugate twin waves(PCTW), orthogonal frequency division multiplexing(OFDM), wave division multiplexing(WDM) and other technologies, this paper uses OptiSystem software to build PCTW-OFDM-WDM system. The mechanism of IQ imbalance caused by probabilistic shaping, crosstalk and statistical correlation noise is analyzed. The theoretical analysis and simulation results show that IQ imbalance is caused by many factors, and only a large IQ imbalance will have a significant impact on the system performance. Geometric shaping scheme has limited effect on IQ imbalance and performance improvement. In the time slot and subcarrier PCTW mode, appropriately changing the location of the distribution of the conjugate components can effectively improve the IQ imbalance and obtain better transmission performance than adjacent structures.