The analysis of the theoretical model of few-mode fiber is not perfect. The channel model of few-mode fiber based on the theory of displacement and rotation of fiber segment is established, which takes into account the factors such as mode crosstalk and differential mode delay. In order to improve the system performance, the influence of probability shaping technology and interleaved encoding technology on signal transmission performance are studied. The simulation results show that after the 16-order QAM (Quadrature Amplitude Modul) signal is transmitted through a few-mode fiber with a length of 50 km, when the forward error correction limit is 3.8×10 ^-3, the optical signal-to-noise ratio is reduced by 3 dB, and the transmission distance is effectively improved.