To improve the quality of the fused image, the study presents a double-branch antagonism network (DANet) for the polarization direction images. The network includes three main modules: feature extraction, fusion, and transformation. First, the feature extraction module incorporates low and high-frequency branches, and the polarization direction images of 0°, 45°, 90°, and 135° are concatenated and imported to the lowfrequency branch to extract energy features. Two sets of polarization antagonism images (0°, 90°, 45°, and 135°) are subtracted and entered into the high-frequency branch to extract detailed features and energy. Detailed features are fused to feature maps. Finally, the feature maps were transformed into fused images. Experiment results show that the fusion images obtained by DANet make obvious progress in visual effects and evaluation metrics, compared with the composite intensity image I, polarization antagonistic image Sd, Sdd, Sh, and Sv, the average gradient, information entropy, spatial frequency, and mean gray value of the image are increased by at least 22.16%, 9.23%, 23.44% and 38.71%, respectively.