In the 100 Gbit/s optical signal transmission system, by comparing the transmission characteristics of the four serial phase modulation formats with those of NRZ-DPSK, RZ-DPSK, NRZ-DQPSK, and RZ-DQPSK, respectively, we show that, when each signal was transmitted over 106 km optical SMF+DCF, the NRZ-DQPSK signal has the highest chromatic dispersion tolerance when the fiber input powers of four phase modulation formats are same.If only considering first order polarization mode dispersion (PMD), RZ-DQPSK has the largest first order PMD tolerance. When adjusting the fiber input power from 0～10 dBm, RZ-DPSK format is very robust against nonlinearities. At last, each phase modulation signal was launched into a third-order Gaussian optical bandpass filter (OBPF) with different bandwidth, and their receipt performance was compared. In addition, the NRZ-DPSK signal has the best transmission performance, if it was launched into a third-order Gaussian optical bandpass filter placed with beyond 125 GHz bandwidth.