The degradation of signal degree of polarization with second order polarization mode dispersion in 40 Gb/s RZ and NRZ optical communication is analyzed by numerical simulation. The results show that the degradation of signal degree of polarization is vibrated by second order polarization mode dispersion. The effect of higher-order polarization mode dispersion on degree of polarization of NRZ and RZ code are also numerically analyzed, the results show that the effect of second polarization mode dispersion on degree of polarization of RZ code are more evident than that on NRZ code. A experiment system is set up to compensate the first and high order polarization mode dispersion effectively by degree of polarization as feedback signal and PSO as compensation arithmetic for RZ and NRZ code system.