Advanced optical modulation format is one of the key technologies of high-speed optical transmission system because of its capability in mitigating transmission impairments, such as dispersion and nonlinear effects. We study the transmission characteristics of non return-to-zero octal differential phase-amplitude-shift keying (NRZ-ODPASK), return-to-zero octal differential phase-amplitude-shift keying (RZ-ODPASK), and carrier-suppressed return-to-zero octal differential phase-amplitude-shift keying (CSRZ-ODPASK) modulation formats for 40 Gbit/s system. It is shown that RZ-ODPASK format can tolerate more chromatic dispersion than the other two formats, especially in its phase channel. Furthermore, for WDM systems, CSRZ-ODPASK format performs better than NRZ-ODPASK and RZ-ODPASK in crosstalk penalty by 0.81 dB and 0.19 dB, respectively. CSRZ-ODPASK is a good choice for high speed optical fiber communication system.