In response to the problem of generating bright spots caused by uneven lighting or smooth reflective materials on the workpiece, a differential feature attention guided polarization image highlight region removal algorithm is proposed, which uses a two-stage network structure to obtain high-quality highlight removal results. The first stage utilizes U-Net network to extract multi-level high-level and low-level features; In the second stage, a differential feature attention guidance module is designed to establish a correlation model between specular reflection and diffuse reflection in the feature space, supplementing and enhancing the original features of the diffuse reflection layer image. In addition, a dataset for polarization highlight removal was produced to verify network performance. Quantitative and qualitative experiments have shown that the proposed algorithm has a better ability to remove highlights than existing algorithms.