Convolutional neural networks (CNNs) stand as pivotal instruments within the realm of deep learning, wielding their influence across an array of domains spanning from computer vision to natural language processing. Their advent has propelled humanity into realms of achievement previously deemed unattainable, facilitating breakthroughs in facial recognition, autonomous vehicle navigation, streamlined retail experiences through self-service supermarkets, and the development of intelligent medical treatment systems. Convolution operation is the core of convolutional neural networks, which is a mechanism empowered by a kernel sliding across input data, thus enabling the extraction of intricate features crucial for pattern recognition. While the weight-sharing property embedded within CNNs efficiently captures local structures within image data, it simultaneously gives rise to a conundrum: computational redundancy. This redundancy manifests prominently due to the overlapping nature of convolution operations, resulting in duplicated multiplications and accumulations for areas where the kernel traverses overlapping segments. This phenomenon not only diminishes computational efficiency but also impairs real-time processing capabilities, underscoring the pressing need for solutions as computational demands surge, imposing formidable challenges upon existing computational hardware platforms.

The efficiency of hardware deployment for convolutions encounters significant obstacles due to inherent redundancy and computational inefficiencies in convolution calculations. This redundancy stems from the overlapping nature of the convolution operation, which entails numerous multiplications and accumulations on identical input samples. This occurs as a small kernel traverses a large dataset, such as an image, performing a series of multiplications at each position and summing their results. The problem arises from the kernel’s overlap with neighboring data segments, resulting in redundant calculations for these overlapping areas. Utilizing multiple devices or clock cycles for these calculations leads to suboptimal resource utilization, thereby limiting real-time processing capabilities. Addressing these challenges is imperative as computational demands escalate, presenting significant hurdles to current computational hardware platforms. In response to the limitations inherent in electronic computation, optical neural networks have emerged as a beacon of promise, poised to redefine the landscape of next-generation computing hardware platforms. Leveraging the expansive bandwidth afforded by photonic chips, optical neural networks can achieve clock frequencies surpassing existing electronic counterparts by orders of magnitude. The various physical dimensions of light, including wavelength, mode, and polarization, offer substantial computational parallelism, leading to a manifold improvement in computational efficiency. An essential advantage of optical computation lies in its propagation-as-computation of light, which allows for ultra-low latency far beyond the capabilities of traditional electronic chips. This unique attribute opens up exciting possibilities for novel applications, such as autonomous driving and ultrafast science.

We summarize and discuss the progress and representative achievements of optical convolutional computing from the perspectives of the definition of convolutional computation and the convolution theorem. Firstly, we introduce the definition of convolutional computation, and based on this, optical convolutions based on dimension interleaving and matrix multiplication are presented. The dimension interleaving scheme (Fig. 3) fully utilizes the high parallelism characteristics of light, greatly improving the efficiency of computing systems. We summarize representative works generated by the four commonly used dimension interleaving methods and unexplored works (Table 1). We also showcase our reflections on this scheme and the proposed work. Due to the two mathematical representations of the matrix multiplication scheme (Fig. 4), it can correspond to spatial projection architecture and on-chip integrated architecture in optical hardware. Starting from basic optical unit devices, we summarize the optical convolution scheme based on matrix multiplication (Fig. 5). Then, we introduce the theorem of convolutional computation, which completes convolution in the transform domain using the Fourier transform. We summarize three types of Fourier transforms used for optical convolution (Fig. 6), namely spatial domain and spatial frequency domain, time domain and frequency domain, vortex beams, and orbital angular momentum. The 4F system scheme was proposed earlier, and subsequently various schemes appeared to optimize it. The frequency domain convolution scheme is newly proposed in the past two years, and we have also contributed some of our ideas and work to this scheme. Subsequently, we introduce the applications of optical convolution (Fig. 8). Currently, most applications focus on the processing of two-dimensional images, and we have also made many contributions in this regard. Optical convolution also has broad application prospects in high-dimensional scenarios. Finally, we provide a summary of the comparison of the above optical convolution schemes in several indicators (Table 2), discussing the overall utilization of computing resources for each scheme, and addressing the redundant issues in convolution.

With the easier manipulation of optical dimensions and the emergence of higher-performance optical devices, the computational efficiency and scale of optical convolutional computing will continue to improve. With the demonstration of more types of optical convolutional computing operations, algorithms, and architectures, optical CNN (OCNN) can serve as a universal building block for various machine learning tasks, potentially bringing revolutionary advances in applications such as autonomous driving, real-time data processing, and medical diagnosis.