The connection between Maxwell’s equations and neural networks opens unprecedented opportunities at the interface between photonics and deep learning. This feature issue highlights recent research progress at the interdisciplinary field of photonics and deep learning and provides an opportunity for different communities to exchange their ideas from different perspectives.

We present a simple approach based on photonic reservoir computing (P-RC) for modulation format identification (MFI) in optical fiber communications. Here an optically injected semiconductor laser with self-delay feedback is trained with the representative features from the asynchronous amplitude histograms of modulation signals. Numerical simulations are conducted for three widely used modulation formats (on–off keying, differential phase-shift keying, and quadrature amplitude modulation) for various transmission situations where the optical signal-to-noise ratio varies from 12 to 26 dB, the chromatic dispersion varies from -500 to 500 ps/nm, and the differential group delay varies from 0 to 20 ps. Under these situations, final simulation results demonstrate that this technique can efficiently identify all those modulation formats with an accuracy of >95% after optimizing the control parameters of the P-RC layer such as the injection strength, feedback strength, bias current, and frequency detuning. The proposed technique utilizes very simple devices and thus offers a resource-efficient alternative approach to MFI.

A high-dimensional quantum key distribution (QKD), which adopts degrees of freedom of the orbital angular momentum (OAM) states, is beneficial to realize secure and high-speed QKD. However, the helical phase of a vortex beam that carries OAM is sensitive to the atmospheric turbulence and easily distorted. In this paper, an adaptive compensation method using deep learning technology is developed to improve the performance of OAM-encoded QKD schemes. A convolutional neural network model is first trained to learn the mapping relationship of intensity profiles of inputs and the turbulent phase, and such mapping is used as feedback to control a spatial light modulator to generate a phase screen to correct the distorted vortex beam. Then an OAM-encoded QKD scheme with the capability of real-time phase correction is designed, in which the compensation module only needs to extract the intensity distributions of the Gaussian probe beam and thus ensures that the information encoded on OAM states would not be eavesdropped. The results show that our method can efficiently improve the mode purity of the encoded OAM states and extend the secure distance for the involved QKD protocols in the free-space channel, which is not limited to any specific QKD protocol.

We propose a plug-and-play (PnP) method that uses deep-learning-based denoisers as regularization priors for spectral snapshot compressive imaging (SCI). Our method is efficient in terms of reconstruction quality and speed trade-off, and flexible enough to be ready to use for different compressive coding mechanisms. We demonstrate the efficiency and flexibility in both simulations and five different spectral SCI systems and show that the proposed deep PnP prior could achieve state-of-the-art results with a simple plug-in based on the optimization framework. This paves the way for capturing and recovering multi- or hyperspectral information in one snapshot, which might inspire intriguing applications in remote sensing, biomedical science, and material science. Our code is available at: https://github.com/zsm1211/PnP-CASSI.

Compressed ultrafast photography (CUP) is the fastest single-shot passive ultrafast optical imaging technique, which has shown to be a powerful tool in recording self-luminous or non-repeatable ultrafast phenomena. However, the low fidelity of image reconstruction based on the conventional augmented-Lagrangian (AL) and two-step iterative shrinkage/thresholding (TwIST) algorithms greatly prevents practical applications of CUP, especially for those ultrafast phenomena that need high spatial resolution. Here, we develop a novel AL and deep-learning (DL) hybrid (i.e., AL+DL) algorithm to realize high-fidelity image reconstruction for CUP. The AL+DL algorithm not only optimizes the sparse domain and relevant iteration parameters via learning the dataset but also simplifies the mathematical architecture, so it greatly improves the image reconstruction accuracy. Our theoretical simulation and experimental results validate the superior performance of the AL+DL algorithm in image fidelity over conventional AL and TwIST algorithms, where the peak signal-to-noise ratio and structural similarity index can be increased at least by 4 dB (9 dB) and 0.1 (0.05) for a complex (simple) dynamic scene, respectively. This study can promote the applications of CUP in related fields, and it will also enable a new strategy for recovering high-dimensional signals from low-dimensional detection.

We demonstrate a smart sensor for label-free multicomponent chemical analysis using a single label-free ring resonator to acquire the entire resonant spectrum of the mixture and a neural network model to predict the composition for multicomponent analysis. The smart sensor shows a high prediction accuracy with a low root-mean-squared error ranging only from 0.13 to 2.28 mg/mL. The predicted concentrations of each component in the testing dataset almost all fall within the 95% prediction bands. With its simple label-free detection strategy and high accuracy, the smart sensor promises great potential for multicomponent analysis applications in many fields.

Particle distribution estimation is an important issue in medical diagnosis. In particular, photon scattering in some medical devices extremely degrades image quality and causes measurement inaccuracy. The Monte Carlo (MC) algorithm is regarded as the most accurate particle estimation approach but is still time-consuming, even with graphic processing unit (GPU) acceleration. The goal of this work is to develop an automatic scatter estimation framework for high-efficiency photon distribution estimation. Specifically, a GPU-based MC simulation initially yields a raw scatter signal with a low photon number to hasten scatter generation. In the proposed method, assume that the scatter signal follows Poisson distribution, where an optimization objective function fused with sparse feature penalty is modeled. Then, an over-relaxation algorithm is deduced mathematically to solve this objective function. For optimizing the parameters in the over-relaxation algorithm, the deep Q-network in the deep reinforcement learning scheme is built to intelligently interact with the over-relaxation algorithm to accurately and rapidly estimate a scatter signal with the large range of photon numbers. Experimental results demonstrated that our proposed framework can achieve superior performance with structural similarity >0.94, peak signal-to-noise ratio >26.55 dB, and relative absolute error <5.62%, and the lowest computation time for one scatter image generation can be within 2 s.

The need for high-speed imaging in applications such as biomedicine, surveillance, and consumer electronics has called for new developments of imaging systems. While the industrial effort continuously pushes the advance of silicon focal plane array image sensors, imaging through a single-pixel detector has gained significant interest thanks to the development of computational algorithms. Here, we present a new imaging modality, deep compressed imaging via optimized-pattern scanning, which can significantly increase the acquisition speed for a single-detector-based imaging system. We project and scan an illumination pattern across the object and collect the sampling signal with a single-pixel detector. We develop an innovative end-to-end optimized auto-encoder, using a deep neural network and compressed sensing algorithm, to optimize the illumination pattern, which allows us to reconstruct faithfully the image from a small number of measurements, with a high frame rate. Compared with the conventional switching-mask-based single-pixel camera and point-scanning imaging systems, our method achieves a much higher imaging speed, while retaining a similar imaging quality. We experimentally validated this imaging modality in the settings of both continuous-wave illumination and pulsed light illumination and showed high-quality image reconstructions with a high compressed sampling rate. This new compressed sensing modality could be widely applied in different imaging systems, enabling new applications that require high imaging speeds.

We propose a practical scheme for end-to-end optical backpropagation in neural networks. Using saturable absorption for the nonlinear units, we find that the backward-propagating gradients required to train the network can be approximated in a surprisingly simple pump-probe scheme that requires only simple passive optical elements. Simulations show that, with readily obtainable optical depths, our approach can achieve equivalent performance to state-of-the-art computational networks on image classification benchmarks, even in deep networks with multiple sequential gradient approximation. With backpropagation through nonlinear units being an outstanding challenge to the field, this work provides a feasible path toward truly all-optical neural networks.

Orbital angular momentum (OAM)-carrying beams have received extensive attention due to their high-dimensional characteristics in the context of free-space optical communication. However, accurate OAM mode recognition still suffers from reference misalignment of lateral displacement, beam waist size, and initial phase. Here we propose a deep-learning method to exquisitely recognize OAM modes under misalignment by using an alignment-free fractal multipoint interferometer. Our experiments achieve 98.35% recognizing accuracy when strong misalignment is added to hyperfine OAM modes whose Bures distance is 0.01. The maximum lateral displacement we added with respect to the perfectly on-axis beam is about &plusmn;0.5 beam waist size. This work offers a superstable proposal for OAM mode recognition in the application of free-space optical communication and allows an increase of the communication capacity.

In this work, we present experimental results concerning excitability in a multiband emitting quantum-dot-based photonic neuron. The experimental investigation revealed that the same two-section quantum dot laser can be tuned through a simple bias adjustment to operate either as a leaky integrate and fire or as a resonate and fire neuron. Furthermore, by exploiting the inherent multiband emission of quantum-dot devices revealed by the existence of multiple lasing thresholds, a significant enhancement in the neurocomputational capabilities, such as spiking duration and firing rate, is observed. Spike firing rate increased by an order of magnitude that leads to an enhancement in processing speed and, more importantly, neural spike duration was suppressed to the picosecond scale, which corresponds to a significant temporal resolution enhancement. These new regimes of operation, when combined with thermal insensitivity, silicon cointegration capability, and the fact that these multiband mechanisms are also present in miniaturized quantum-dot devices, render these neuromorphic nodes a proliferating platform for large-scale photonic spiking neural networks.

We demonstrate a neural network capable of designing on-demand multiple symmetry-protected bound states in the continuum (BICs) in freeform structures with predefined symmetry. The latent representation of the freeform structures allows the tuning of the geometry in a differentiable, continuous way. We show the rich band inversion and accidental degeneracy in these freeform structures by interacting with the latent representation directly. Moreover, a high design accuracy is demonstrated for arbitrary control of multiple BIC frequencies by using a photonic property readout network to interpret the latent representation.

The connection between Maxwell&rsquo;s equations and artificial neural networks has revolutionized the capability and efficiency of nanophotonic design. Such a machine learning tool can help designers avoid iterative, time-consuming electromagnetic simulations and even allows long-desired inverse design. However, when we move from conventional design methods to machine-learning-based tools, there is a steep learning curve that is not as user-friendly as commercial simulation software. Here, we introduce a real-time, web-based design tool that uses a trained deep neural network (DNN) for accurate far-field radiation prediction, which shows great potential and convenience for antenna and metasurface designs. We believe our approach provides a user-friendly, readily accessible deep learning design tool, with significantly reduced difficulty and greatly enhanced efficiency. The web-based tool paves the way to present complicated machine learning results in an intuitive way. It also can be extended to other nanophotonic designs based on DNNs and replace conventional full-wave simulations with a much simpler interface.

A new approach to optical fiber sensing is proposed and demonstrated that allows for specific measurement even in the presence of strong noise from undesired environmental perturbations. A deep neural network model is trained to statistically learn the relation of the complex optical interference output from a multimode optical fiber (MMF) with respect to a measurand of interest while discriminating the noise. This technique negates the need to carefully shield against, or compensate for, undesired perturbations, as is often the case for traditional optical fiber sensors. This is achieved entirely in software without any fiber postprocessing fabrication steps or specific packaging required, such as fiber Bragg gratings or specialized coatings. The technique is highly generalizable, whereby the model can be trained to identify any measurand of interest within any noisy environment provided the measurand affects the optical path length of the MMF&rsquo;s guided modes. We demonstrate the approach using a sapphire crystal optical fiber for temperature sensing under strong noise induced by mechanical vibrations, showing the power of the technique not only to extract sensing information buried in strong noise but to also enable sensing using traditionally challenging exotic materials.

We propose a modified supervised learning algorithm for optical spiking neural networks, which introduces synaptic time-delay plasticity on the basis of traditional weight training. Delay learning is combined with the remote supervised method that is incorporated with photonic spike-timing-dependent plasticity. A spike sequence learning task implemented via the proposed algorithm is found to have better performance than via the traditional weight-based method. Moreover, the proposed algorithm is also applied to two benchmark data sets for classification. In a simple network structure with only a few optical neurons, the classification accuracy based on the delay-weight learning algorithm is significantly improved compared with weight-based learning. The introduction of delay adjusting improves the learning efficiency and performance of the algorithm, which is helpful for photonic neuromorphic computing and is also important specifically for understanding information processing in the biological brain.

As artificial neural networks (ANNs) continue to make strides in wide-ranging and diverse fields of technology, the search for more efficient hardware implementations beyond conventional electronics is gaining traction. In particular, optical implementations potentially offer extraordinary gains in terms of speed and reduced energy consumption due to the intrinsic parallelism of free-space optics. At the same time, a physical nonlinearity—a crucial ingredient of an ANN—is not easy to realize in free-space optics, which restricts the potential of this platform. This problem is further exacerbated by the need to also perform the nonlinear activation in parallel for each data point to preserve the benefit of linear free-space optics. Here, we present a free-space optical ANN with diffraction-based linear weight summation and nonlinear activation enabled by the saturable absorption of thermal atoms. We demonstrate, via both simulation and experiment, image classification of handwritten digits using only a single layer and observed 6% improvement in classification accuracy due to the optical nonlinearity compared to a linear model. Our platform preserves the massive parallelism of free-space optics even with physical nonlinearity, and thus opens the way for novel designs and wider deployment of optical ANNs.

Over the past decades, photonics has transformed many areas in both fundamental research and practical applications. In particular, we can manipulate light in a desired and prescribed manner by rationally designed subwavelength structures. However, constructing complex photonic structures and devices is still a time-consuming process, even for experienced researchers. As a subset of artificial intelligence, artificial neural networks serve as one potential solution to bypass the complicated design process, enabling us to directly predict the optical responses of photonic structures or perform the inverse design with high efficiency and accuracy. In this review, we will introduce several commonly used neural networks and highlight their applications in the design process of various optical structures and devices, particularly those in recent experimental works. We will also comment on the future directions to inspire researchers from different disciplines to collectively advance this emerging research field.

We report an approach assisted by deep learning to design spectrally sensitive multiband absorbers that work in the visible range. We propose a five-layered metal-insulator-metal grating structure composed of aluminum and silicon dioxide, and we design its structural parameters by using an artificial neural network (ANN). For a spectrally sensitive design, spectral information of resonant wavelengths is additionally provided as input as well as the reflection spectrum. The ANN facilitates highly robust design of a grating structure that has an average mean squared error (MSE) of 0.023. The optical properties of the designed structures are validated using electromagnetic simulations and experiments. Analysis of design results for gradually changing target wavelengths of input shows that the trained ANN can learn physical knowledge from data. We also propose a method to reduce the size of the ANN by exploiting observations of the trained ANN for practical applications. Our design method can also be applied to design various nanophotonic structures that are particularly sensitive to resonant wavelengths, such as spectroscopic detection and multi-color applications.

Intelligent coding metasurface is a kind of information-carrying metasurface that can manipulate electromagnetic waves and associate digital information simultaneously in a smart way. One of its widely explored applications is to develop advanced schemes of dynamic holographic imaging. By now, the controlling coding sequences of the metasurface are usually designed by performing iterative approaches, including the Gerchberg&ndash;Saxton (GS) algorithm and stochastic optimization algorithm, which set a large barrier on the deployment of the intelligent coding metasurface in many practical scenarios with strong demands on high efficiency and capability. Here, we propose an efficient non-iterative algorithm for designing intelligent coding metasurface holograms in the context of unsupervised conditional generative adversarial networks (cGANs), which is referred to as physics-driven variational auto-encoder (VAE) cGAN (VAE-cGAN). Sharply different from the conventional cGAN with a harsh requirement on a large amount of manual-marked training data, the proposed VAE-cGAN behaves in a physics-driving way and thus can fundamentally remove the difficulties in the conventional cGAN. Specifically, the physical operation mechanism between the electric-field distribution and metasurface is introduced to model the VAE decoding module of the developed VAE-cGAN. Selected simulation and experimental results have been provided to demonstrate the state-of-the-art reliability and high efficiency of our VAE-cGAN. It could be faithfully expected that smart holograms could be developed by deploying our VAE-cGAN on neural network chips, finding more valuable applications in communication, microscopy, and so on.

Super-resolution structured illumination microscopy (SR-SIM) provides an up to twofold enhanced spatial resolution of fluorescently labeled samples. The reconstruction of high-quality SR-SIM images critically depends on patterned illumination with high modulation contrast. Noisy raw image data (e.g., as a result of low excitation power or low exposure time), result in reconstruction artifacts. Here, we demonstrate deep-learning based SR-SIM image denoising that results in high-quality reconstructed images. A residual encoding&ndash;decoding convolutional neural network (RED-Net) was used to successfully denoise computationally reconstructed noisy SR-SIM images. We also demonstrate the end-to-end deep-learning based denoising and reconstruction of raw SIM images into high-resolution SR-SIM images. Both image reconstruction methods prove to be very robust against image reconstruction artifacts and generalize very well across various noise levels. The combination of computational image reconstruction and subsequent denoising via RED-Net shows very robust performance during inference after training even if the microscope settings change.

Deep learning in the context of nano-photonics is mostly discussed in terms of its potential for inverse design of photonic devices or nano-structures. Many of the recent works on machine-learning inverse design are highly specific, and the drawbacks of the respective approaches are often not immediately clear. In this review we want therefore to provide a critical review on the capabilities of deep learning for inverse design and the progress which has been made so far. We classify the different deep-learning-based inverse design approaches at a higher level as well as by the context of their respective applications and critically discuss their strengths and weaknesses. While a significant part of the community&rsquo;s attention lies on nano-photonic inverse design, deep learning has evolved as a tool for a large variety of applications. The second part of the review will focus therefore on machine learning research in nano-photonics “beyond inverse design.” This spans from physics-informed neural networks for tremendous acceleration of photonics simulations, over sparse data reconstruction, imaging and “knowledge discovery” to experimental applications.

All-optical binary convolution with a photonic spiking vertical-cavity surface-emitting laser (VCSEL) neuron is proposed and demonstrated experimentally for the first time, to the best of our knowledge. Optical inputs, extracted from digital images and temporally encoded using rectangular pulses, are injected in the VCSEL neuron, which delivers the convolution result in the number of fast (<100 ps long) spikes fired. Experimental and numerical results show that binary convolution is achieved successfully with a single spiking VCSEL neuron and that all-optical binary convolution can be used to calculate image gradient magnitudes to detect edge features and separate vertical and horizontal components in source images. We also show that this all-optical spiking binary convolution system is robust to noise and can operate with high-resolution images. Additionally, the proposed system offers important advantages such as ultrafast speed, high-energy efficiency, and simple hardware implementation, highlighting the potentials of spiking photonic VCSEL neurons for high-speed neuromorphic image processing systems and future photonic spiking convolutional neural networks.

Imaging through scattering media is one of the hotspots in the optical field, and impressive results have been demonstrated via deep learning (DL). However, most of the DL approaches are solely data-driven methods and lack the related physics prior, which results in a limited generalization capability. In this paper, through the effective combination of the speckle-correlation theory and the DL method, we demonstrate a physics-informed learning method in scalable imaging through an unknown thin scattering media, which can achieve high reconstruction fidelity for the sparse objects by training with only one diffuser. The method can solve the inverse problem with more general applicability, which promotes that the objects with different complexity and sparsity can be reconstructed accurately through unknown scattering media, even if the diffusers have different statistical properties. This approach can also extend the field of view (FOV) of traditional speckle-correlation methods. This method gives impetus to the development of scattering imaging in practical scenes and provides an enlightening reference for using DL methods to solve optical problems.

Imaging through nonstatic scattering media is one of the major challenges in optics, and encountered in imaging through dense fog, turbid water, and many other situations. Here, we propose a method to achieve single-shot incoherent imaging through highly nonstatic and optically thick turbid media by using an end-to-end deep neural network. In this study, we use fat emulsion suspensions in a glass tank as a turbid medium and an additional incoherent light to introduce strong interference noise. We calibrate that the optical thickness of the tank of turbid media is as high as 16, and the signal-to-interference ratio is as low as -17 dB. Experimental results show that the proposed learning-based approach can reconstruct the object image with high fidelity in this severe environment.

Being invisible at will has been a long-standing dream for centuries, epitomized by numerous legends; humans have never stopped their exploration steps to realize this dream. Recent years have witnessed a breakthrough in this search due to the advent of transformation optics, metamaterials, and metasurfaces. However, the previous metasurface cloaks typically work in a reflection manner that relies on a high-reflection background, thus limiting the applications. Here, we propose an easy yet viable approach to realize the transmitted metasurface cloak, just composed of two planar metasurfaces to hide an object inside, such as a cat. To tackle the hard-to-converge issue caused by the nonuniqueness phenomenon, we deploy a tandem neural network (T-NN) to efficiently streamline the inverse design. Once pretrained, the T-NN can work for a customer-desired electromagnetic response in one single forward computation, saving a great amount of time. Our work opens a new avenue to realize a transparent invisibility cloak, and the tandem-NN can also inspire the inverse design of other metamaterials and photonics.

Structural color based on Fabry&ndash;Perot (F-P) cavity enables a wide color gamut with high resolution at submicroscopic scale by varying its geometrical parameters. The ability to design such parameters that can accurately display the desired color is therefore crucial to the manufacturing of F-P cavities for practical applications. This work reports the first inverse design of F-P cavity structure using deep learning through a bidirectional artificial neural network. It enables the production of a significantly wider coverage of color space that is over 215% of sRGB with extremely high accuracy, represented by an average &Delta;E2000 value below 1.2. The superior performance of this structural color-based neural network is directly ascribed to the definition of loss function in the uniform CIE 1976-Lab color space. Over 100,000 times improvement in the design efficiency has been demonstrated by comparing the neural network to the metaheuristic optimization technique using an evolutionary algorithm when designing the famous painting of “Haystacks, end of Summer” by Claude Monet. Our results demonstrate that, with the correct selection of loss function, deep learning can be very powerful to achieve extremely accurate design of nanostructured color filters with very high efficiency.

While deep learning has demonstrated tremendous potential for photonic device design, it often demands a large amount of labeled data to train these deep neural network models. Preparing these data requires high-resolution numerical simulations or experimental measurements and cost significant, if not prohibitive, time and resources. In this work, we present a highly efficient inverse design method that combines deep neural networks with a genetic algorithm to optimize the geometry of photonic devices in the polar coordinate system. The method requires significantly less training data compared with previous inverse design methods. We implement this method to design several ultra-compact silicon photonics devices with challenging properties including power splitters with uncommon splitting ratios, a TE mode converter, and a broadband power splitter. These devices are free of the features beyond the capability of photolithography and generally in compliance with silicon photonics fabrication design rules.

Speed, generalizability, and robustness are fundamental issues for building lightweight computational cameras. Here we demonstrate generalizable image reconstruction with the simplest of hybrid machine vision systems: linear optical preprocessors combined with no-hidden-layer, “small-brain” neural networks. Surprisingly, such simple neural networks are capable of learning the image reconstruction from a range of coded diffraction patterns using two masks. We investigate the possibility of generalized or “universal training” with these small brains. Neural networks trained with sinusoidal or random patterns uniformly distribute errors around a reconstructed image, whereas models trained with a combination of sharp and curved shapes (the phase pattern of optical vortices) reconstruct edges more boldly. We illustrate variable convergence of these simple neural networks and relate learnability of an image to its singular value decomposition entropy of the image. We also provide heuristic experimental results. With thresholding, we achieve robust reconstruction of various disjoint datasets. Our work is favorable for future real-time low size, weight, and power hybrid vision: we reconstruct images on a 15 W laptop CPU with 15,000 frames per second: faster by a factor of 3 than previously reported results and 3 orders of magnitude faster than convolutional neural networks.

Optical focusing through scattering media is of great significance yet challenging in lots of scenarios, including biomedical imaging, optical communication, cybersecurity, three-dimensional displays, etc. Wavefront shaping is a promising approach to solve this problem, but most implementations thus far have only dealt with static media, which, however, deviates from realistic applications. Herein, we put forward a deep learning-empowered adaptive framework, which is specifically implemented by a proposed Timely-Focusing-Optical-Transformation-Net (TFOTNet), and it effectively tackles the grand challenge of real-time light focusing and refocusing through time-variant media without complicated computation. The introduction of recursive fine-tuning allows timely focusing recovery, and the adaptive adjustment of hyperparameters of TFOTNet on the basis of medium changing speed efficiently handles the spatiotemporal non-stationarity of the medium. Simulation and experimental results demonstrate that the adaptive recursive algorithm with the proposed network significantly improves light focusing and tracking performance over traditional methods, permitting rapid recovery of an optical focus from degradation. It is believed that the proposed deep learning-empowered framework delivers a promising platform towards smart optical focusing implementations requiring dynamic wavefront control.