An adaptive polarization mode dispersion (PMD) compensation experiment is reported in a 40-Gb/s phase shaped binary transmission (PSBT) communication system, with the use of a new digital signal processor (DSP)-based optical PMD compensator. PMD tolerance is found to be enhanced by 8 ps after PMD compensation with 1-dB optical signal-to-noise ratio (OSNR) penalty. Under the condition of fast change of states of polarization up to 85 rad/s in the fiber link, the performance of our PMD compensator undergoes the bit error ratio (BER) test for as long as 10 h.