In order to study dispersion compensation of optical differential phase shift keying (DPSK) modulation format in high-speed optical fiber transmission system, dispersion compensation principle of dispersion compensation fiber was used to compensate the dispersion of 40Gbit/s optical fiber transmission system. The spectrum characteristics of three DPSK modulation formats in 40Gbit/s single channel optical fiber transmission system were analyzed. The dispersion tolerance of three codes was simulated. When considering the nonlinearity of optical fibers, dispersion compensation schemes of three modulation schemes were simulated. The results show that, optical non-return-to-zero differential phase shift keying (NRZ-DPSK) signal has the best dispersion tolerance, but it is greatly affected by non-linearity. 33% return-to-zero differential phase shift keying signal has poor dispersion tolerance, but the effect of dispersion compensation is better than that of NRZ-DPSK. Carrier-suppressed return-to-zero differential phase shift keying signals can suppress both dispersion and nonlinearity. Three DPSK modulation schemes have the best dispersion compensation effect in symmetrical compensation scheme 2. This simulation study has reference significance for dispersion compensation of optical DPSK signal in optical fiber.