Fig. 1. Structure diagram of convolutional neural network (CNN)

Fig. 2. Optical convolution principle based on dimensional interweaving

Fig. 3. Major advances in optical convolution schemes based on dimensional interweaving. (a) The first scheme to achieve optical convolution based on time-wavelength interweaving; (b) on-chip integrated scheme for optical convolution based on time-wavelength interweaving^[66]; (c) compact optical convolution processing unit based on multimode interference^[67]; (d) optical convolution scheme based on time-space interweaving^[68]; (e) optical convolution scheme based on space-wavelength interweaving proposed by author’s team of this paper; (f) envisaged implementation scheme for optical convolution based on space-mode interweaving

Fig. 4. Two forms of transformation for two-dimensional convolution. (a) Model transformed into generalized matrix multiplication (GeMM); (b) model of two-dimensional (2D) convolution; (c) model transformed into one-dimensional (1D) convolution

Fig. 5. Optical convolution basic units and related technical schemes based on matrix multiplication. (a) Basic unit of spatial projection architecture; (b) optical convolution scheme based on Dammann grating^[70]; (c) optical convolution scheme based on micro-lens array (MLA)^[72]; (d) basic unit of on-chip integrated architecture; (e) optical convolution scheme based on rectangular MZI network^[78]; (f) optical convolution scheme based on triangular MZI network^[79]; (g) optical coherent dot product chip^[80]; (h) optical convolution scheme based on micro-ring resonator (MRR) weight banks; (i) small MRR array proposed by our team for performing large complex matrix-vector multiplication^[81]; (j) optical convolution based on mode division multiplexing using MRR^[76]; (k) optical convolution scheme based on waveguide modulator (WG) array^[53]; (l) optical convolution scheme based on phase change materials^[45]; (m) multi-mode optical convolution scheme based on metasurfaces^[77]

Fig. 6. Common Fourier transform pairs of Fourier form optical convolution computation. (a) Spatial frequency domain and spatial domain; (b) time domain and frequency domain; (c) vortex beam and orbital angular momentum (OAM)

Fig. 7. Main advances in optical convolution schemes based on Fourier transform. (a) Earliest proposed optical convolution scheme based on 4F system^[109]; (b) optical convolution scheme based on metamaterial lens^[118]; (c) lensless optical convolution scheme^[123]; (d) on-chip 4F system optical convolution scheme based on metasurface^[121]; (e) on-chip 4F system optical convolution scheme based on star couplers^[115]; (f) optical convolution scheme for complex values based on OAM^[131]; (g) optical frequency-domain convolution scheme based on dual-frequency comb^[125]; (h) optical frequency-domain convolution scheme based on fiber ring modulator^[128]; (i) optical frequency-domain convolution scheme based on lithium niobate modulator; (j) optical frequency-domain convolution scheme based on four-wave mixing in highly nonlinear fiber^[132]

Fig. 8. Applications of 2D optical convolution. (a) Image processing using metalens^[118]; (b) cell segmentation using Fourier space diffraction neural networks^[124]; (c) face recognition using lensless opto-electronic diffraction neural networks^[123]; (d) human emotion recognition using on-chip microring weight banks^[141]