Research background

In the 1990s, computer science researchers began using digital image processing to replace optical processing, achieving superior visual effects in images, which is named as Computational Photography. Later, the integration of optics, information, and computation has been further developed and refined, leading to the emergence of computational imaging technology, a cutting-edge field in imaging today. Computational imaging revisited the imaging process from an information perspective, thoroughly transforming the way visual information is captured, processed, and interpreted.

Recently, Prof. Xiaopeng Shao, Deputy Director of the Xi'an Institute of Optics and Precision Mechanics of CAS, along with Associate Researcher Jinpeng Liu and Prof. Fei Liu from Xidian University, focused on the development of computational imaging technology, published a review paper titled "Future-proof Imaging: Computational Imaging" in Advanced Imaging. The review comprehensively summarizes computational imaging technology from multiple perspectives, including development history, essential concepts, and definitions. Starting from the optical imaging process, the paper categorizes computational imaging technology into computational light sources, computational media, computational optical systems, computational detectors, and computational processing. The paper thoroughly analyzes the acquisition and utilization methods of light field information throughout the imaging chain, systematically reviews the research framework of key technologies in computational imaging, and discusses current challenges and future development directions.

Contents

This review introduces the development of optical imaging from two aspects, light manipulation and light perception, analyses the physical significance of different light field projections, and summarizes the characteristics of traditional photoelectric imaging models. According to the characteristics of the whole-link optimization of computational imaging technology, the review starts from the modulation and decoding technologies of computational light source, computational medium, computational optical system, computational detector and computational processing, introduces the imaging principles, advantages and disadvantages of each technology, and summarizes the current major applications of computational imaging.

1. Computational light source

In dark environments, human visual capability significantly decreases, making it necessary to use additional lighting to clearly recognize the surroundings. In such cases, lighting plays a crucial role in enhancing the signal-to-noise ratio of imaging. Similarly, by appropriately controlling the light source, it is possible to enhance or even surpass imaging performance metrics. Computational light source involves encoding the spatial, temporal, and physical dimensions at the light source of the imaging system. The higher the dimensions of encoding, the greater the freedom to modulate the illumination light field, which is central to computational imaging—solving problems that lower dimensions cannot handle by increasing dimensionality. Given the current state of technology, light intensity modulation serves as the foundation for all modulation methods, as all modulations ultimately manifest as changes in the light intensity detected by the sensor. Therefore, this section analyzes the imaging technologies that have emerged from various illumination control methods, including phase modulation, wavelength modulation, light vector modulation, time modulation, and coherent imaging, all based on light intensity modulation.

Fig. 1 Computational light source

2. Computational medium

When light passes through complex transmission media, especially scattering media, the target information becomes hidden within the scattered light field. The difficulty of interpreting the original light field information is directly determined by the scattering medium. To some extent, the scattering medium functions as a complex amplitude modulation matrix in imaging (which can sometimes be simplified to a phase modulation matrix). On one hand, calibrating the medium's modulation matrix can eliminate the effects of the scattering medium and extract the target information. On the other hand, by incorporating additional dimensions of physical information—such as phase, polarization, and multispectral data—into the intensity information, it is possible to separate and interpret the target information embedded in the scattered light field. This section provides a deep analysis of the scattering process of target information within the medium and discusses various techniques for recovering target information from two perspectives: separating scattered light and utilizing scattered light.

Fig.2 Computational medium

2. Computational optical system

Computational optical system is designed for efficient information transmission, not only collecting sufficient light energy but also ensuring an adequate flux of target information through light field modulation. These systems optimize the interplay between optical hardware and software algorithms, achieving high-quality imaging while reducing precision constraints and hardware complexity. This section analyzes and summarizes the characteristics and applications of various computational optical technologies, including metalenses, wide-field optical systems, minimalistic optical systems, adaptive optics, and single-pixel imaging. After breaking away from traditional design approaches, computational optical systems are evolving toward smaller sizes, simpler structures, lower costs, wider fields of view, higher resolutions, stronger environmental adaptability, and higher-dimensional information acquisition.

Fig.3 Computational optical system

3. Computational detector

A computational detector is designed for computational imaging applications, equipped with the capability to detect spatial, temporal, and physical dimensions. Traditional photodetectors obtain image information by detecting the energy of photons, but the signal response mode based on the photoelectric effect cannot accurately describe the multidimensional physical information of the light field, leading to significant information loss and difficulty in reconstruction. To address the limitations of current detection modes, this section reviews a range of designs and applications in the field of computational detectors, including non-uniform detectors, curved surface detectors, multi-dimensional physical quantity detectors, and ultra-high-speed detectors, based on different detection purposes.

Fig.4 Computational detector

4. Computational processing

As the last link in the imaging chain, computational processing involves using algorithms to refine the image data captured by detectors. This process extracts valuable target information from the image data, enhancing details, improving contrast, increasing resolution, and widening the field of view. This section categorizes computational processing into three main aspects based on the purpose of data processing: image fusion, image enhancement, and super-resolution, and then provides a detailed analysis of the principles and applications of each technology.

Fig.5 Computational processing

5. Summary and prospect

In summary, computational imaging technology has achieved a seamless integration of optical imaging and information computational processing, delivering revolutionary advantages that traditional optical imaging techniques find hard to reach. Driven by information transmission, this technology adopts an integrated, end-to-end design approach, enabling higher-dimensional light field information and enhancing the utilization and interpretability of light field data. This results in disruptive imaging effects that are "higher, farther, broader, smaller, and stronger." Although computational imaging technology has rapidly emerged, gradually transforming the traditional landscape of visual science and demonstrating the limitless possibilities of imaging technology, it still faces numerous challenges in the future, such as the foundational theories of computational imaging, the design of orthogonal projections for light field information, and multidimensional optoelectronic detectors. As advancements in algorithms and hardware continue to progress, computational imaging technology is poised to create an even more vivid and impactful "visual world," profoundly influencing our life and work.