Due to the bit correlation of high-order modulation,polarization code can not effectively polarize the channel in optical communication system.In this paper,a polarization interleaver for polarization multiplexing optical communication system is proposed.The proposed interleaver uses a unified encoder to encode the information bits of the system,and then interleaves the coded sequence,which breaks the bit correlation of high-order modulation,reduces the impact on the channel polarization of the polarization code,and improves the performance of the communication system.At the same time,the 2D and 4D modulation and coding schemes are compared and analyzed,and the scaling factor is added in the decoding symbol metric calculation to improve the bit error rate (BER) performance of the system.Simulation results show that compared with the traditional bit interleaved coded modulation (BICM),the BER performance of the polarization interleaver is improved by about one order of magnitude when the channel is additive white Gaussian noise (AWGN),and the BER performance of the system is improved by about 0.5 dB when the scaling factor is added.In DP-16QAM coherent optical communication system,the performance of 4D polarization interleaving scheme is improved by about 1.3 dB compared with the traditional BICM scheme when the BER is 10－4.Moreover,the tolerance of 4D modulation scheme to polarization mode dispersion (PMD) is better than that of 2D modulation scheme under different PMD.