The interference caused by clouds and the surrounding environment leads to significant distortions in the atmospheric polarization data over large areas. We propose an atmospheric polarization mode reconstruction method based on non-neighborhood constraints to address this issue. First, this method leverages the fact that the distribution of atmospheric polarization is solely dependent on the position of the Sun during the same measurement to obtain ground truth data affected by interference through the control variable approach. Second, considering the continuous spatial distribution of atmospheric polarization modes, our approach incorporates a non-neighborhood feature restoration module to uncover dependencies between non-neighborhood regions within the atmospheric polarization data. This capability enables the reconstruction of measured atmospheric polarization data even in case of extensive distortion across large areas. Finally, the results of our ablation experiment show that the indicators peak signal-to-noise ratio and structural similarity are increased by 17% and 25% compared with the non-neighborhood constraint module when the cloud area reaches about 60%.