Fig. 1. Synergy of hardware physics and software algorithms in optical neural networks

Fig. 2. Optical neural networks based on noise-aware training. (a) Free-space optical vector dot product units with limited bit depth through quantization-aware training methods^[30]; (b) ultra-low energy optical neural network inference and training processes based on physical noise models^[31]; (c) diffractive neural networks with inter-layer-misalignment-aware training^[32]; (d) diffractive neural networks with inter-layer-translation-, rotation-, and scaling-aware training^[33]

Fig. 3. Optical neural networks based on in-situ training methods. (a) On-chip diffractive neural networks capable of processing multimodal data^[41]; (b) in-situ training with power awareness^[45]; (c) in-situ training of on-chip three-layer deep optical neural networks^[46]

Fig. 4. Training optical neural networks using direct feedback alignment. (a) Training procedures of on-chip linear layers, after completing the forward computation of the neural network and obtaining the prediction error, the gradient information for optimizing the weights is derived by multiplying this prediction error with a random matrix and incorporating the derivative of the activation function used during the forward computation^[49]; (b) augmented direct feedback alignment training method with enhanced applicability range, where it no longer relies on the accurate derivative of the activation function used during the forward computation, but instead employs a different nonlinear activation function^[50]

Fig. 5. On-chip optical neural networks with pruning. (a) Optical linear layer designed using low-rank matrix decomposition^[51]; (b) pruning of MZI arrays implementing unitary transformations, where different parameter subsets within the array differentially impact the transformation^[52]; (c) pruning method based on sparse activation values^[53]; (d) hardware-aware pruning method based on the lottery ticket hypothesis^[55]

Fig. 6. Applications of other algorithms in optical neural networks. (a) Significant enhancement of the expressive power of linear diffractive neural networks using knowledge distillation techniques^[57]; (b) complex-valued stochastic dropout in diffractive neural networks^[60]; (c) improved predictive robustness of diffractive neural networks through ensemble learning^[62]; (d) performance enhancement of diffractive neural networks using low-rank adaptation-based fine-tuning methods^[64]

Fig. 7. Optoelectronic generative neural networks. (a) Utilizing optoelectronic random number generators as noise sources for on-chip generative neural networks^[35]; (b) large-scale optoelectronic generative neural networks constructed by integrating multiple light scattering and diffractive neural networks^[67]; (c) photonic probabilistic neurons based on controllable quantum fluctuation probability distributions^[70]

Fig. 8. Training neural networks utilizing the physical characteristics of optical systems. (a) Training optical neural networks using linear symmetric optical systems^[82]; (b) training deep neural networks with random projections from multiple light scattering^[83]

Fig. 9. New applications enabled by the synergy of physical characteristics of optical computing systems and algorithms. (a) Hidden object sensing and detection^[86]; (b) orbital angular momentum spectrum reconstruction^[87]; (c) spin and orbital angular momentum spectra reconstruction^[88]; (d) accelerating decision-making problems using optical spatiotemporal chaos^[89]