Infrared imaging systems often produce fringe noise in imaging results owing to the non-uniformity of the detection unit. To obtain better correction results, most deep learning-based infrared image nonuniformity correction algorithms adopt complex network structures, which increase the computational cost. This study proposes a lightweight network-based infrared image non-uniformity correction algorithm and designs a lightweight multi-scale downsampling module (LMDM) for the encoding process of the Unet network. The LMDM uses pixel splitting and channel reconstruction to realize feature map downsampling and realizes multi-scale feature extraction using multiple cascaded depth-wise separable convolutions (DSC). In addition, the algorithm introduces a lightweight channel attention mechanism for adjusting feature weights to achieve better contextual information fusion. The experimental results show that the proposed algorithm reduces memory use by more than 70% and improves the processing speed of the infrared images by more than 24% compared with the comparison algorithm while ensuring that the corrected image has a clear texture, rich details, and sharp edges.