This study introduced a differential polarization division multiplexing (DPDM) scheme for all-fiber, high-accuracy spectral measurement for the first time. Launching two orthogonal linearly polarized light beams into multimode fibers, the DPDM elegantly reconstructs the input spectra from the differential speckle, which is a more comprehensive spectral-to-spatial encoding with fewer noises and better contrast. A spectral measurement with 2 pm resolution and 2,000 channels was achieved. Compared with traditional transmission matrix and polarization division multiplexing schemes, the proposed DPDM approach reduced the spectral reconstruction error by 83% and 55%, respectively, offering a simple and effective solution for highly accurate spectral measurements.