Infrared and Laser Engineering, Volume. 53, Issue 5, 20240049(2024)
Deep learning-based infrared imaging degradation model identification and super-resolution reconstruction
Junfeng Cao1,2,3,4, Qinghai Ding5, Depeng Zou6, Hengjia Qin6, and Haibo Luo1,2,3
Author Affiliations
1Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences, Shenyang 110016, China2Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China3Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110169, China4University of Chinese Academy of Sciences, Beijing 100049, China5Space Star Technology Co., Ltd., Beijing 100086, China6The Third Military Representative Office of the Air Force Equipment Department in Shenyang, Shenyang 110016, Chinashow less
ObjectiveThe limited resolution of infrared devices, constrained by cost and manufacturing technology, remains a challenge. While deep learning-based single image super-resolution (SISR) has shown promise in enhancing image resolution, its application in real-world infrared images is hindered by the complexity of actual degradation, including spatial non-uniform blur caused by optical aberration and assembly error, as well as variations in the blur kernel due to environmental temperature changes. A deep learning-based approach for infrared imaging degradation model identification and super-resolution reconstruction is proposed to tackle these challenges. This method entails solving the degradation model using a convolutional neural network to describe the evolution of blur kernels, along with a super-resolution reconstruction method that adheres to the constraints of the degradation model and incorporates online learning of degradation parameters.
MethodsImages of calibration targets are captured using an infrared camera placed in a high and low temperature chamber, along with a portable target simulator placed outside it (Fig.1-2). These images are utilized to calibrate the blur kernels. A convolutional neural network (CNN) is employed to construct a model that characterizes the relationship between blur kernel, pixel coordinate, and operating temperature (Fig.3). The model is trained using the calibrated blur kernels. Additionally, a super-resolution network is developed and trained (Fig.4). The operating temperature is initially estimated using the low-resolution image. Next, the initial blur kernels are estimated by inputting the operating temperature into the kernel model. Subsequently, super-resolution reconstruction is conducted based on the estimated blur kernels, and the reconstructed image is utilized to refine the operating temperature and blur kernel estimation. Iterative processes improve the accuracy of blur kernel estimation, leading to enhanced reconstruction outcomes.
Results and DiscussionsThe blur kernels of the infrared imaging system exhibit significant variation in response to temperature changes and spatial locations (Fig.6). The trained blur kernel model effectively predicts blur kernels using temperature and pixel coordinate inputs (Fig.7). The average PSNR between predicted and actual blur kernels across different operational temperatures is consistently high, with a minimum of 32.2 dB and an average of 37.1 dB, indicating precise predictions (Fig.8). The calibration and modeling of blur kernels provide valuable prior information for super-resolution reconstruction, resulting in enhanced reconstruction outcomes. Consequently, the proposed algorithm produces visually appealing results with improved detail (Fig.10-11) and enhances objective quality evaluation metrics such as the natural image quality evaluator (NIQE), perception-based image quality evaluator (PIQE), and blind/referenceless image spatial quality evaluator (BRISQUE) (Tab.1).
ConclusionsA novel approach is proposed for infrared super-resolution imaging, including degradation model identification and iterative super-resolution reconstruction. The degradation model is based on a convolutional neural network and is solved using offline calibration data. It can predict blur kernels across various temperatures and spatial positions, reducing the need for extensive calibration work. Online degradation parameter correction is achieved through an iterative optimization network alternating between estimating the blur kernel and reconstructing the super-resolution image. By leveraging the degradation model, the complex high-dimensional blur kernel estimation problem is simplified into a low-dimensional operating temperature estimation problem, streamlining the solution process. Through iterations, the accuracy of blur kernel estimation improves, leading to superior super-resolution reconstruction outcomes. Experimental results demonstrate that calibrating and modeling blur kernels enhance prior information for super-resolution reconstruction, yielding superior results. Additionally, the proposed method adapts to a wider temperature range, reducing the stringency of athermalization design requirements for infrared optical systems.