In the coherent receiver of polarization division multiplexing (PDM) optical communication system, the modules of polarization demultiplexing and carrier phase recovery are required. Extended Kalman filter is used into polarization division multiplexing-16 quadrature amplitude modulation (PDM-16QAM) coherent receiver to achieve polarization and phase tracking quickly and accurately. In the single channel 112 Gb/s PDM-16QAM optical communication system, the simulation results show that the maximum polarization rotation rate of extended Kalman filter tracking is nearly 100 times faster than that of Cascade multimode algorithm. Meanwhile, Kalman filter also has a high accuracy, and the convergence speed and accuracy can be controlled by adjusting the tuning parameters.Specifically, only 0.2 dB sensitivity penalty at the bit error rate (BER) limit of 10-3 is induced when extended Kalman filter is applied on the signal with 100 kHz line width and 18 Mrad/s polarization rotation. A long-haul transmission simulation is raised to test the performance of extended Kalman filter.