Traditional imaging technology, based on direct intensity detection, is constrained by the intrinsic mechanisms of the photoelectric effect and detector fabrication processes. This presents significant bottlenecks in achieving key performance indicators such as high resolution, high sensitivity, and a high signal-to-noise ratio. In particular, the spatial resolution and space-bandwidth product of the acquired image have long been limited by the detector’s pixel size and physical device scale. In recent years, computational imaging has emerged as a new paradigm that integrates front-end spatio-temporal encoding and modulation with back-end digital computational inversion. It offers innovative solutions to overcome the numerous limitations of traditional imaging techniques and represents the future direction of advanced optical imaging. We systematically review the research progress of pixel super-resolution imaging techniques based on spatio-temporal encoding. Focusing on modulation strategies at the aperture and focal planes, we elucidate the mechanisms of high-dimensional light field modulation and their corresponding back-end image reconstruction algorithms. These two aspects work in synergy to break the detector’s Nyquist sampling limit, enabling the effective decoupling and restoration of sub-pixel spatial information. While maintaining system compactness and high light-throughput efficiency, these technologies significantly enhance the spatial resolution and space-bandwidth product of the imaging system, providing novel solutions and key technological support for demanding applications such as long-range, wide-field-of-view, and high-precision optoelectronic detection and recognition in complex environments.

Modulation of amplitude or phase at the aperture plane is one of the key technical paths to achieving pixel super-resolution imaging. By introducing a programmable or fixed mask pattern at the aperture, the wavefront information is modulated, thereby changing the system’s point spread function. This allows high-frequency information to be encoded in a specific manner and multiplexed within the sampling process of multiple low-resolution frames. Combined with reconstruction algorithms, a high-resolution image that surpasses the Nyquist sampling limit can be recovered from this encoded low-resolution data, effectively enhancing the spatial resolution of the imaging system. Compared to amplitude encoding, phase encoding is a modulation scheme with a higher signal-to-noise ratio. A phase mask modulates the wavefront phase by introducing spatially varying optical path differences without attenuating the incident light energy. Therefore, it can achieve a higher image signal-to-noise ratio under the same conditions as amplitude encoding. Metasurfaces, with their sub-wavelength unit structures, leverage specific physical mechanisms to precisely and discretely manipulate the phase, amplitude, and polarization of the light field, enabling wavefront modulation at an ultra-thin scale that is difficult for traditional optical elements to achieve. Their essence is to directly expand the available degrees of freedom of the system at the aperture plane, reallocating the transmission channel capacity within the framework of “space-bandwidth product control” to realize wavefront encoding-decoding multiplexing. Since the size of their unit structures is comparable to or even smaller than a detector pixel, these devices are a natural fit for encoding strategies that enhance spatial resolution, such as “pixel-level encoding” and “sub-pixel displacement,” providing a novel hardware platform for breaking the Nyquist sampling limit. The aforementioned encoding devices at the aperture plane, whether traditional amplitude/phase encoders or new devices like metasurfaces, all offer feasible paths to increase the effective Nyquist sampling rate and reduce system volume, holding the promise of further unleashing the potential of pixel super-resolution within future spatio-temporal encoding computational imaging frameworks.

In the computational optical imaging chain, the focal plane is the critical location where the photodetector performs photoelectric conversion, transforming an optical image into a digital signal. Through fine-grained modeling and control of the detector’s spatial sampling characteristics, spatio-temporal encoding techniques at the focal plane can encode and reconstruct sub-pixel level light field information via on-chip and pixel-level modulation at the end of the imaging chain. Classic focal-plane spatio-temporal encoding techniques for pixel super-resolution include micro-scanning. This technique introduces sub-pixel relative displacements between images on the system’s focal plane and uses the detector to record the corresponding intensity information at different sub-pixel sampling positions at different time, thereby achieving spatio-temporal modulation. This sequence of spatio-temporally encoded low-resolution images implicitly contains high-frequency information beyond the Nyquist sampling limit, providing crucial prior information for subsequent super-resolution reconstruction. Based on the driving method used to generate the sub-pixel shifts, micro-scanning is typically categorized into active and passive techniques. Beyond the technical route that relies on the micro-motion of a single sensor to acquire sub-pixel displacements, another parallel and efficient strategy is the use of camera array systems. By integrating multiple low-resolution detector units, these systems can simultaneously capture multiple low-resolution images of the same scene in a single exposure. Due to the fixed physical spatial offset between the imaging units, the captured frames contain inherent sub-pixel relative shift information. In addition to using micro-scanning devices, spatio-temporal encoding can also be achieved using programmable devices such as digital micromirror devices (DMDs) or spatial light modulators (SLMs). These devices project a series of pre-set or dynamically generated spatially structured encoding patterns onto an intermediate image plane or a plane conjugate to the object/image, enabling precise encoding modulation. The dynamic programmability of these devices allows for real-time and point-by-point changes to the amplitude and phase of the light field at the focal plane, enabling sophisticated optical encoding of the target scene and increasing the overall information content and efficiency of the imaging system’s encoding.

Alternatively, a fixed encoding mask can be placed in front of the focal plane to encode the incident light field. Compared to programmable active devices, it is passive and low-power, has a simpler system structure, and possesses the potential for real-time imaging. However, while on-chip encoding modulation can effectively enhance the system’s spatial resolution, it inherently does so by reducing the detector’s light energy efficiency in exchange for improved information acquisition in the spatial dimension. To avoid the light-throughput loss introduced by encoding masks, modulated images with a high signal-to-noise ratio can be obtained by altering the detector’s spatial sampling method. Compared to traditional sensors with a regular and periodic layout, constructing a detector with a non-periodic and translationally asymmetric sampling grid allows for the sampling of more diverse spatial frequency components during image acquisition, providing a new means of information modulation for super-resolution imaging.

Overall, pixel super-resolution imaging technology based on spatio-temporal encoding and modulation breaks the limitations of traditional imaging in resolution, sensing dimensions, and information throughput by leveraging flexible encoding in the physical domain and efficient decoding in the digital domain. This is essentially a strategy of “information exchange” or “space-bandwidth product control,” where, in adherence to physical laws, information in certain dimensions is sacrificed to enhance performance in target dimensions or to enable the perception of new dimensions. Looking forward, “new architectures, new devices, and new methods” will mutually reinforce one another. Through joint multi-dimensional information encoding and intelligent decoding, they will further expand the boundaries of perception, enabling the capture of higher-dimensional and finer information about the objective world and advancing imaging technology from “seeing clearly” to “understanding”. Although challenges in data processing, algorithm generalizability, system complexity, and cost still lie ahead, computational imaging, by virtue of its unique “physics-computation” synergy, is certain to exhibit ever-broader application prospects in numerous fields, including scientific research, industrial inspection, medical diagnostics, consumer electronics, and even national security, and it will continue to drive the iterative innovation of optoelectronic imaging technology.