To overcome challenges arising from inaccurate polarization normal gradients and difficulties in obtaining real three-dimensional (3D) information in scene polarization 3D imaging—attributed to factors like uneven illumination, complex colors, materials, and changes in observation direction under a large field of view—a new approach utilizing a direction-aware convolution neural network is explored. The method involves constructing a scene depth estimation network with direction perception abilities, correcting polarization normal gradients using the convolutional neural network estimated scene depth, and ultimately reconstructing the 3D image through a gradient-based integration algorithm. Experimental results showcase this approach's effectiveness in resolving azimuth ambiguity inherent in polarization, enhancing normal gradient accuracy in scenes with uneven illumination and wide field of view, and successfully restoring the real 3D shape of the scene while preserving intricate texture details. The findings affirm the efficacy and superiority of the proposed technology.