Traditional scene-based non-uniformity correction algorithms generally suffer from non-uniformity residuals and ghosts. In view of this, we propose an infrared image adaptive algorithm based on the encoding and decoding residual network. The algorithm focuses on the characteristics of the adaptive correction problem. Following the UNet structure, the residual image is generated through multiscale sampling and learning residual mapping. Batch normalization and PReLU are used to improve the correction effect. Finally, the global skip connection is used to obtain the final correction result. The experimental results of correcting the simulated non-uniform infrared image sequence and the real infrared image sequence showed that this method improved the objective data of the peak signal to noise ratio (PSNR) and roughness, compared with existing non-uniformity correction algorithms. Moreover, the subjective visual effect was clearer, and the degree of detail retention was high.