Transmission performance of six modulation formats is numerically compared in 40*40-Gb/s wavelength division multiplexing (WDM) repeaterless systems. The results indicate that return-to-zero differential phase shift keying (RZ-DPSK) (33% duty cycle) demonstrates the best performance among all the formats when the accumulated amplified spontaneous emission power (AASEP) is small. But with the increasing of AASEP, the superiority of RZ-DPSK transmission performance over the other five modulation formats is vanished. AASEP is justified to be the major source for the deterioration of the superiority.