Fig. 1. Schematic of the synergy between photonics and AI.

Fig. 2. Artificial intelligence for photonic devices.

Fig. 3. ANN modeling. (a) Artificial neuron structure, (b) ANN model structure, and (c) photonic devices are described by two types of labels: physical variables $x$ and physical responses $y$.

Fig. 4. Different neural network architectures. (a) Tandem networks: these consist of several modules connected in series, and the different modules are connected to each other through an intermediate layer to form an overall network structure. (b) CNNs: these consist of multiple convolutional, pooling, and fully connected layers. The convolutional layer extracts the local features of the image, the pooling layer is used to reduce the dimensionality and enhance the generalization ability of the model, and the fully connected layer maps the extracted features to the output of the final task. (c) GANs: these consist of a generator and a discriminator. Generator to generate fake data, discriminator to determine the authenticity of the data; the two, through the confrontation training, are constantly optimized, and finally, the generator can generate samples that are very similar to the real data. (d) Variational autoencoders: these consist of an encoder, which maps the input data to a probability distribution in the latent space, and a decoder, which reconstructs the data from the samples in the latent space. (e) Physics-informed neural networks: PINNs fit input–output relationships through neural networks while embedding physical equations (e.g., partial differential equations, initial and boundary conditions) as constraint terms in the loss function. During the training process, the network uses physical constraints to guide learning, realizing the integration of data-driven and physical models.

Fig. 5. Application of deep-learning methods. (a) Metamaterials: demonstrate the process of metamaterial image evolution during a certain number of training steps.⁶⁵ (b) Photonic crystal: mode switching among different bulk modes in a topologically trivial lattice designed by an ANN.⁶⁶ (c) Nanoparticles: simultaneous inverse design of structural parameters and material information of core-shell nanoparticles from given electric and magnetic dipoles extinction spectra using deep learning.⁶⁷ (d) Microwave cloak: at 8.2-GHz frequency, the reflection spectrum shows that the spectrum predicted based on ANNs matches well the real spectrum obtained by simulation.⁶⁸ (e) Optical storage: sketches of different geometric models encoding 2, 3, 4, or 5 bit sequences using ANNs to store the encoded information.⁶⁹ (f) Soliton microcomb: second-order and higher-order dispersion is obtained from the target microcomb using the Lugiato–Lefever equation and genetic algorithm, and the microcavity geometry is obtained using a pretrained forward DNN coupled with GA.⁷⁰ (g) Silicon color design: schematic of silicon nanostructures and generated colors.⁷¹ (h) Grating coupler: schematic diagram of the grating coupler structure, in which the guided light incident from the left is vertically diffracted by a column with a periodic staggered height of 220 nm and a grating with an L-shaped cross section partially etched to 110 nm.⁷² (i) Power splitter: forward and inverse modeling of nanophotonic devices using deep-learning networks, which can take the device topology design as input and the spectral response of components as labels and vice versa.⁷³ (j) Plasmonic nanodimers: based on the analysis of Born–Kuhn-type plasma nanodimers, neural networks capable of successfully predicting chiral properties and further inverse design of the plasma structure to achieve the desired circular dichroism were designed.⁷⁴ (k) Optical switch: all-optical plasma switches use neural networks to predict spectra through hidden layers after inputting geometric details.⁷⁵

Fig. 6. Typical examples of nanophotonic devices based on deep-learning methods. (a) 3D chiral metamaterial: schematic of designed 3D chiral metamaterials and their predicted reflection and circular dichroism spectra.¹⁰⁵ (b) Topology-optimized metasurface: schematic diagram of metasurface inverse design based on training of the GAN and topology optimization. The generated devices can be fed back to the neural network for retraining and optimization.⁹⁹ (c) Power splitter: inverse design of power splitter based on GAN combined with simulation neural network and self-attention mechanism.¹²⁵

Fig. 7. Applications of PINN in nanophotonics. (a) Schematic of a PINN for solving inverse problems in photonics based on partial differential equations.⁹⁴ (b) PINN reconstruction of the dielectric constant profile from a data set of known scattered field profiles.⁹⁴ (c) Schematic of the auxiliary PINNs solution to the radiative transfer theory problem.¹¹⁵ (d) Contours for finite-element method forward scattering simulations, inversion results for the complex dielectric function, real and imaginary parts of the complex electric field $E_z$, and the complex electric field $E_z$ reconstructed from PINNs.¹¹⁰

Fig. 8. (a) Flow chart of the gradient-based inverse design algorithm. (b) Flow chart of the adjoint method.

Fig. 9. Nanophotonic device by gradient-based inverse design. (a) Spatial mode multiplexer: optimal design patterns and simulated field ($E_y$) evolution for spatial pattern multiplexer.¹⁴⁵ (b) Power splitter: scanning electron microscopy (SEM) image of the fabricated broadband $1\times 3$ power splitter and the electromagnetic energy density in the device at 1550 nm.¹⁴³ (c) Wavelength demultiplexer: simulated electromagnetic energy density of a three-channel wavelength multiplexer at three operating wavelengths.¹⁴⁹ (d) Grating couplers based on diamond design: inverse-designed vertical coupler with analog field superimposed in red.¹⁵⁶ (e) Fano resonators: SEM image of a cascaded Fano–Lorentzian resonator. The enlarged image shows the reflector designed in inverse direction on the silicon waveguide in the resonator-waveguide coupling region.¹⁵¹ (f) Grating couplers: the electric field in the structure of the grating coupler with a target bandwidth of 120 nm is simulated at 1550 nm.¹⁵⁴ (g) SPIN software optimization process: (1) continuous optimization; (2) discretization; (3) discrete optimization. Fabrication constraints are enforced at this time.¹⁵⁷ (h) Metalens: the metalenses are illuminated by normally incident $x$-polarized plane waves. The incident field outside the aperture of the metalens is blocked by a layer of perfect electrical conductors.¹⁵⁹

Fig. 10. (a) Flow chart of the variable density method. (b) Flow chart of the level set method. (c) Flow chart of the bidirectional evolutionary structure optimization.

Fig. 11. Nanophotonic devices by the gradient-based inverse design. (a) Spatial mode multiplexer.¹⁴⁴ (b) Inverse design results (silicon regions are shown in black and silica regions in white). (c) Optical microscope image of the final fabricated device. (d) Experimentally measured $S$ parameters of the back-to-back test structure. (Shaded areas indicate the minimum and maximum values from three different measured devices from three dies, and solid lines indicate the average values.) (e) Three-channel wavelength demultiplexer.¹⁴⁴ (f) Inverse design results. (g) Optical microscope image of the final fabricated device. (h) Experimentally measured $S$ parameters. (i) Three-way power splitter.¹⁴⁴ (j) Inverse design results. (k) Optical microscope image of the final fabricated device. (l) Experimentally measured $S$ parameters (dashed line indicates perfect 1/3 beam splitting ratios). (m) SEM image of the inverse designed-fixed coupler.¹⁴⁷ (n) Schematic diagram of the computing platform, consisting of input generator, photonic processor, and complex output.¹⁴⁷ (o) Optical microscope image of photonic platform.¹⁴⁷ (p) Photograph of the photonic platform and wire bonding [the red square marks one platform detail in panel (o)].¹⁴⁷

Fig. 12. (a) Flow chart of the GA. (b) Coding method.

Fig. 13. Nanophotonic device designed based on GA. (a) Polarization route: SEM image of a $970\mathrm{nm}\times 1240\mathrm{nm}$ polarization router.¹⁹⁷ (b) Metasurface absorber: schematic of the optimized binary pattern ${\text{A}}_0$ in the crystal cell and SEM image of the pattern ${\text{A}}_0$ array.¹⁹³ (c) Chiral plasmonic metasurface: top view of design pattern A and SEM image of chiral metasurface.¹⁹⁴ (d) Broadband absorption optimization: structural schematics and absorption spectra of the different generations.¹⁹⁵ (e) Metasurface design: different combinations of coefficients on the pattern of light produced.¹⁹⁶ (f) Optical frequency microcombs: SEM image of the photonic-crystal resonators. The inset on the right highlights a section of the chirped corrugation.¹⁹⁸

Fig. 14. (a) Crossover operator and (b) variation operator.

Fig. 15. (a) PSO iteration process. (b) Flow chart of PSO.

Fig. 16. Nanophotonic device designed based on PSO. (a) Power splitter: binary particle swarm optimized $2\times 2$ power splitter.¹⁹⁹ (b) Nanosensor: schematic diagram of a nanosensor consisting of periodic gold nanoridges.²⁰¹ (c) Optical coupler: structure of the proposed multisegment directional coupler.²⁰⁰ (d) Photonic crystal: simulation results of p-polarized incident wave.²⁰⁴ (e) Varifocal lens: schematic of the varifocal lens. The inset shows a single-cell sample.²⁰²

Fig. 17. (a) Flow chart of the simulated annealing algorithm. Nanophotonic device based on simulated annealing algorithm optimized design. (b) Metasurface: simulated near-electric field distribution under $x$-polarized normal incidence.²³³ (c) Spin Hall device: schematic of an on-chip broadband photonic spin element, where the incident light is coupled into different waveguides according to its spin states.²³⁴

Fig. 18. (a) Flow chart of hill-climbing algorithm. Optimized design of nanophotonic device based on hill-climbing algorithm. (b) Graphene metasurfaces: structure of the first optimized metasurface.²³⁸ (c) One-dimensional photonic crystal split-beam nanocavity: schematic diagram of symmetrical cavity design.²³⁹

Fig. 19. Direct binary search flow chart.

Fig. 20. Optimized design of nanophotonic devices based on direct binary search. (a) Mode converter: optimized layout of ${\mathrm{TE}}_1-{\mathrm{TE}}_0$ mode converters and optimized optical field distribution for mode-order converter.²⁵² (b) Power splitter: SEM image of the entire manufacturing facility consisting of a dual-mode 3 dB power divider and three mode multiplexers.²⁵⁵ (c) Polarization splitter-rotator: ${\mathrm{TM}}_0-{\mathrm{TE}}_0$ mode simulated light field, ${\mathrm{TM}}_0-{\mathrm{TE}}_0$ mode cross-sectional light field at input and output ports.²⁵⁶

Fig. 21. (a) Tabu search flow chart. Nanophotonic device based on tabu search optimized design. (b) Polarization filters based on photonic lattices: optimized holes-in-slab configuration (57 scatterers).²⁴³ (c) Beam shaping of 2D photonic lattices: photonic lattice used for the beam-shaping problem. The dashed line indicates the plane used to calculate the desired beam.²⁴⁴

Fig. 22. (a) Network architecture for phase unwrapping.²⁸⁷ (b) One quantitative phase image of multiple lung cancer cells. The images are focused manually and then unwrapped by the quality-guided unwrapping algorithm. The unwrapped focused-phase images are used for labeled training in the model. The cross section and 3D representation of one cell with wrapped and unwrapped signals are shown.²⁸⁸ (c) The DNN blindly outputs artifact-free phase and amplitude images of the object using only one hologram intensity. This DNN is composed of convolutional layers, residual blocks, and upsampling blocks and rapidly processes a complex-valued input image in a parallel, multiscale manner.²⁸⁹ (d) (i) The intensity data are captured by illuminating the sample from different angles with an LED array. (ii) Training CNN to reconstruct high-resolution phase images. The input to the CNN is low-resolution intensity images; the output of the CNN is the ground-truth phase image reconstructed using the traditional FPM algorithm. The network is then trained by optimizing the network’s parameters that minimize a loss function calculated based on the network’s predicted output and the ground truth. (iii) The network is fully trained using the first data set at 0 min and then can be used to predict phase videos of dynamic cell samples frame by frame.²⁹⁰

Fig. 23. Examples of network structure for AI-assisted polarization imaging. (a) Architectures of polarization denoising residual dense network (PDRDN) and residual dense block (RDB).³⁰⁴ (b) Architecture of FIPNet, which consists of three parts: feature extraction layer, fusion layer, and reconstruction layer.³⁰⁵ (c) A reflection separation network takes a cascaded architecture with three modules: semireflector orientation estimation, polarization-guided separation, and separated layers refinement.³⁰⁶ (d) A network tailored to polarization-based dehazing pipeline, which consists of two stages: transmitted light estimation and original scene radiance reconstruction.³⁰⁷ (e) A network with multibranch architecture to handle different hierarchical inputs. The physics-based prior confidence map for the weighted fusion of different inputs and the self-supervised AoLP loss to force the network to learn the prior knowledge between the normal and AoLP.³⁰⁸

Fig. 24. AI-assisted snapshot compact SI. (a)–(d) Results of the spectral combining of the AI reconstruction and the DOE design with diffractive rotation.³²⁹ (a) The fabricated DOE that generates spectrally varying PSFs for SI. Inset: a camera installed with the DOE. (b) The PSFs at different wavelengths. (c) Overview of the network architecture. (d) The RGB image of a reconstructed SI and the comparison between the reconstructed spectrum and the ground truth of point 1 in the scene. (e)–(g) Results of the shift-variant color-coded diffractive SI system.³³³ (e) Optimization of the optical elements is carried out using an end-to-end AI approach. (f) RGB image of a reconstructed hyperspectral image and the comparison between the reconstructed spectrum and the ground truth of point 1 in the scene. SCCD types 1 to 3 denote three different types of CCA utilized in the system. Spiral denotes a system without CCA. (h)–(j) Different types of pixelated filter array: (h) Fabry–Perot filter;³³⁵ (i) freeform-shaped metasurface filter;³³⁶ (j) film filter.³³⁷ (k)–(m) Results of computational SI with CMOS-compatible random array of Fabry–Perot filters shown in panel (h).³³⁵ (k) Performance of hyperspectral image reconstruction simulated for three hyperspectral image data sets, including the RGB show of reconstruction and the error map between the reconstruction and the ground truth. (l) Experimental results of the SI for a standard color sample. (m) The dependence of the frame rate on the image resolution for AI-based reconstruction and the iterative reconstruction with 50 iteration steps.

Fig. 25. Heat-assisted detection and ranging (HADAR) with AI-assisted decomposition.³⁴⁰ (a) Pipeline of HADAR: HADAR takes thermal photon streams as input, records hyperspectral-imaging heat cubes, addresses the ghosting effect through AI-assisted TeX decomposition, and generates TeX vision for improved detection and ranging. (b) TeX vision demonstrated on the database and the outdoor experiments, showing that HADAR sees textures through the darkness with a comprehensive understanding of the scene. (c)–(h) Ranging based on the raw thermal images (c), (d), AI reconstructed images in the HADAR technique at night (e), (f) and daylight RGB vision (g), (h).

Fig. 26. AI-assisted end-to-end platform for digital pathology using hyperspectral autofluorescence microscopy and deep-learning-based virtual histology.³⁴³ (a) Automated workflow with virtual staining and AI scoring that mimics the current pathology workflow. (b)–(e) Classical H&E stained images (b) or the immunofluorescence images [(c) elastin + $\alpha$-SMA, (d) nuclei, and (e) CD]⁶⁸ of a tissue slice. (f)–(i) Images of the adjacent slice generated by a linear projection of the autofluorescence spectral image with different channel-related weights to enhance different components [(f) a uniform projection mimicking the autofluorescence intensity imaging result, (g) extracellular matrix, (h) nuclei, (i) macrophages]. (j) Neural network architecture of the generator of virtual stainer. AF, autofluorescence; BF, bright field. (k) BF real and virtual images stained with H&E. (l)–(o) Correlation of the slide level nonalcoholic steatohepatitis feature attributes predicted by segmentation models on real stains versus virtual stains [(l) percent steatosis, (m) percent lobular inflammation, (n) log-normalized hepatocyte balloon count, (o) fibrosis density].

Fig. 27. Schematic diagram of RNN. (a) Traditional neural network architecture with input, hidden, and output layers. (b) RNN architecture and an unfolding structure with $t$ time steps. $X(t)$: input state. $h(t)$: hidden state. $o(t)$: output state. $W_1$, $W_2$, and $W_r$ represent input, output, and recurrent weight matrices, respectively. (c) LSTM cell architecture with forget, input, output, and cell states.

Fig. 28. Functions of RNN in nonlinear compensation for optical communication. (a) Schematic diagram of LSTM based on sliding window.³⁵⁴ The autoencoder is represented by the blocks Tx BRNN, channel, and Rx BRNN. (b) The principle of Bi-RNN models.³⁵⁵ The Bi-RNN model processes distorted symbols with intersymbol dependencies to estimate bitwise BER, optimizing complexity, and performance for 16-QAM and 32-QAM. (c) Architecture of LSTM combined with CNN for nonlinear compensation.³⁵⁶ The feature maps $y_f$ from the convolutional layer are fed into either two dense layers (forming the CNN + MLP structure, with the number of layers determined by the Bayesian optimizer) or a single Bi-LSTM layer.

Fig. 29. Various optical-sensing applications implemented using LSTM. (a) LSTM-CNN model for vibration sensing.³⁷⁶ The optical cable is installed directly above the PCCP pipe and fixed with fixtures. Different signals exhibit distinct characteristics across the frequency band and more pronounced local features in the time-frequency domain. Based on LSTM and CNN architectures, a neural network was designed using time-domain waveforms along with their DWT and STFT as inputs. This integrated feature set enables effective pattern recognition. (b) Optical fiber sensing based on the LSTM-CNN model in the surgery.³⁷⁷ The LSTM-CNN framework is utilized to process perioperative heart rate (HR) and respiratory rate (RR) frequency signals. Trends are extracted from HR and RR, whereas CNN and LSTM are employed for feature extraction and processing, respectively. (c) Crowded abnormal scene detection using Bi-LSTM and CNN.³⁷⁸ The proposed methodology utilizes optical flow features to capture frame-level spatial information. Temporal information across the data set is modeled using a Bi-LSTM. The key components of the proposed architecture include constructing an optical feature matrix, integrating a CNN with a Bi-LSTM, and implementing a novel inference mechanism.

Fig. 30. Matrix computation using an MZI mesh. (a) Legend for interpreting the symbols used in other subgraphs. Two predominant methods are illustrated: (b) the Reck scheme³⁸⁸ and (c) the Clement scheme.³⁸⁹ The left side of the figure displays the spatial layout of the MZIs, with the number in each yellow block indicating the order of light manipulation by each MZI. The red dashed arrows denote the sequence for decomposing the unitary matrix. The colors blue and green surrounding the red arrows indicate column and row eliminations, respectively. The right side of the figure shows the corresponding elimination order of unitary matrix elements. (d) MZI mesh for universal complex-valued matrix through SVD decomposition.

Fig. 31. Various photonic circuits designed for matrix-vector multiplication. (a) Micrograph of a photonic circuit engineered to compute unitary matrices.³² Different methods for realizing real-valued matrix computations through coherent MZI mesh structures are shown: (b) using an incoherent laser source with power detection³⁵ and (c) constructing the real part of a unitary matrix.³⁹¹

Fig. 32. Self-configuring strategies in optical systems. (a) A self-aligning universal beam coupler.^393,394 (b) Application of the ratio method for calibrating triangular meshes.³⁹⁵ (c), (d) Use of the reversed local light interference method to calibrate universal feedforward meshes.^396,397

Fig. 33. Some gradient-free calibration methods. (a) Execution process of GA.³⁹⁸ (b) Whole pipeline for MZI mesh calibration using GA.³⁹⁹ (c) Bacterial foraging training algorithm is implemented on MZI mesh.⁴⁰⁰

Fig. 34. In situ training method is proposed to realize the BP algorithm in photonic circuits. (a) Procedure of in situ training⁴⁰⁴ and (b) experimental verification for in situ training.⁴⁰⁵

Fig. 35. Incoherent optical computing circuit architectures. (a) A $4\times 4$ nonnegative matrix is realized using a microring array.⁴⁰⁸ (b) In the microring array, output power in both the through port and drop port is detected to realize real-valued matrix computation.⁴⁰⁹ (c) A recursive structure named SDDLN is used to realize matrix-vector multiplication.⁴¹⁰

Fig. 36. Some advances for recent optical computing circuits. The first column [(a), (b)] shows fault-tolerance computing architecture: (a) stacked FFT,⁴¹⁵ (b) redundant rectangular mesh and permuting rectangular mesh.⁴¹⁷ The second column [(c), (d)] shows some miniaturization strategies for computing devices: (c) 3D arrangement of MZI mesh for matrix computation,⁴¹⁸ (d) PBWs are instead of MZI as programmable units to minimize the footprint.⁴¹⁹ The third column [(e)–(g)] demonstrates that the computing parallelism can be enlarged via WDM,⁴²⁰ FDM,⁴⁰⁷ and MDM⁴²¹ technologies.

Fig. 37. All-optical convolution using a $4f$-system under various configurations: coherent light sources in panels (a)⁴³⁰ and (b)⁴³¹ and incoherent light sources in panels (c)⁴³² and (d).⁴³³ Panels (a) and (c) utilize amplitude-only masks, whereas panels (b) and (d) employ phase-only masks.

Fig. 38. All-optical differentiator (a)–(c) and integrator (d)–(f) based on compact resonance structures. The phase-shifted Bragg grating can be designed to realize optical (a) differentiation⁴³⁵ and (d) integration.⁴⁴² (b), (e) Ruan et al. theoretically demonstrated differentiation and integration can be reconfigured in the same device by controlling the propagating loss of surface plasmon polariton.⁴³⁶ (c) Experimental realization of optical differentiation on surface plasmonic structure.⁴³⁷ (f) Integration is presented using a dielectric slab.⁴⁴¹

Fig. 39. Free-space optical matrix-vector multiplier. (a) Schematic diagram for matrix-vector multiplication proposed by Goodman.⁴²⁶ (b) Convolution realization through two metasurfaces.⁴⁴⁵ (c) Coherent system for realizing matrix computation.⁴⁴⁶ (d) Matrix-vector multiplier applied to imaging sensing for optical encoding.³⁸² (e) Experimental verification of dot product operation close to the shot-noise limit of detected photons.⁵⁶ (f) CMOS-compatible matrix processor supporting large input vector size.⁴⁴⁷ (g) Spatial-temporal multiplexed matrix computing system, where matrix elements and input vector are encoded via VCSEL arrays, exhibiting efficient electro-optic conversion and compact footprint.⁴⁴⁸

Fig. 40. Training methods for ${\mathrm{D}}^2\mathrm{NN}$. (a) In situ training procedure of ${\mathrm{D}}^2\mathrm{NN}$ includes four steps: FP, error calculation, BP, and gradient update.⁴⁵² (b) The flow chart for dual adaptive training method.⁴⁰³ (c) The data flow for physics-aware training.⁴⁵³ (d) The conceptual illustration for hybrid training of the optical neural network.⁴⁵⁴

Fig. 41. (a)–(c) Types of light sources used in ${\mathrm{D}}^2\mathrm{NN}$, including (a) monochromatic light source,¹³ (b) spatially incoherent monochromatic light source,⁴⁵⁸ and (c) broadband pulse source.⁴⁵⁹ (d)–(f) Types of ${\mathrm{D}}^2\mathrm{NN}$ structures, including (d) Fourier-space diffractive DNN,⁴⁹ (e) ensemble learning of diffractive neural network,⁴⁶⁰ and (f) diffractive network in network and diffractive RNN.⁴⁵⁵

Fig. 42. Diffracted layers are miniaturized by reducing working wavelength or designing on-chip diffracted structures. (a) Fabrication procedure of germanium-based diffraction grating.⁴⁶² (b) Optical machine-learning decryptor is physically 3D printed by galvo-dithered two-photon nanolithography, and integrated with a CMOS chip.⁴⁶³ (c) Exploded schematic diagram of metasurface-based diffractive neural network integrated with a CMOS chip.⁴⁶⁴ (d) Scanning electron microscope image of an on-chip metalens.⁴⁶⁵ (e) Schematic of on-chip DONN. The diffractive unit composed of three identical silicon slots is used to modulate the amplitude and phase of the optical wave.⁴⁶⁶ (f) The electric field distribution (left) and refractive index distribution (right) of the coherent photonic device that performs unitary matrix computation.⁴⁶⁷ (g) Schematic of metastructures in a SiPh platform using an inverse-design method based on the effective index approximation with low-index contrast constraint.⁴⁶⁸

Fig. 43. High-parallelism ${\mathrm{D}}^2\mathrm{NN}$ inference using (a) polarization multiplexing,⁴⁷⁵ (b) wavelength multiplexing,⁴⁷⁶ and (c) OAM multiplexing⁴⁷⁷ technologies.

Fig. 44. AI-related applications for all-optical ${\mathrm{D}}^2\mathrm{NN}$. (a) Handwritten digit recognition.⁵³ (b) Fashion product recognition.⁵³ (c) Video-based human action recognition.⁴⁵⁵ (d) Image reconstruction.⁴⁷⁸ (e) Subwavelength phase imaging.⁴⁷⁹ (f) All-optical image encryption using incoherent illumination.⁴⁶¹ (g) Superresolution display.⁴⁸⁰ (h) All-optical decryptors using coherent illumination.⁴⁶³

Fig. 45. Hybrid opto-electrical computing system empowers the machine-vision field. (a) Handwritten digit recognition through optical-digital implementation.⁴³² (b) Malaria parasite detection using learned sensing network.⁴⁸³ (c) Imaging compression using a multiply scattering medium and reconstruction by sparse optimization techniques.⁴⁸⁴ (d) End-to-end computational camera design paradigm to realize achromatic extended depth of field.⁴⁸⁵ (e) Joint optimization of microscope point spread function and differentiable reconstruction algorithm to achieve 3D information reconstruction.⁴⁸⁶ (f) The flow chart for depth map estimation using a phase-coded aperture camera.⁴⁸⁷

Fig. 46. Recent high-performance optical computing chips to support advanced AI tasks. (a) The data flow of the all-analog photoelectronic chip, which can support energy-efficient and ultrahigh-speed vision tasks.⁴⁸⁹ (b), (c) Large-scale photonic chiplets are proposed to deploy large models for AGI tasks⁴⁹⁰ such as (b) music generation and (c) image generation.

Fig. 47. (a) Structure of an LNOI modulator. (b) Modulation depth with voltage.³⁷² (c) Architecture of an SOA-based neural network.⁴¹

Fig. 48. Several representative works of PCMs as a nonvolatile memory and weight element. (a) A waveguide-integrated PCM metasurface.⁵¹³ (b) A PCM-integrated cross-bar array for parallel convolution.⁴⁰⁶ (c) A PCM pad array as the neural synapse.⁵¹⁰ (d) A PCM integrated all-optical abacus.⁵¹²

Fig. 49. Nonlinear activation units. (a) A Ge on Si nonlinear activation unit structure and (b) its nonlinear response curve.⁵²³ (c) An image recognition neural network with a quantum nonlinear dot activation layer and (d) a ReLU-like response with a quantum dot activation unit.⁴⁴