In order to improve the spectrum utilization of fiber communication system, the dense wavelength division multiplexing (DWDM) system of 8-level amplitude and differential phase-shift keying (8APSK) and differential 8-level phase-shift keying (D8PSK) are set up. Simulation results show that when the probe channel is 8APSK, the optimal incident average optical power increases with increasing of 8APSK amplitude modulation index. Whether it is 8APSK or D8PSK modulation, four-wave mixing noise has little influence on the system performance when the incident average optical power is optimal. While it exceeds the optimal value, the influence increases rapidly. For the discussed DWDM system, the transmission performance of 8APSK signal with amplitude modulation index optimization is obviously better than that of D8PSK, which is mainly attributed to the better tolerance of total nonlinearities for the former.