Significance With the advent of the period of big data, deep learning is playing a very important role in daily life and scientific research, and it has been widely applied in image processing, speech recognition, autonomous driving, and other fields. Deep learning in the optical field, as a data-driven algorithm, can effectively improve computational efficiency and imaging quality, approaching or even breaking through physical limits. An artificial neural network is a basic form of imitating biological neurons and the working principle of the human brain to complete learning process of internal principle or to extract target features. Different kinds of complicated neural networks are put forward and geared to the demands of different application scenarios for the target that often requiring different network architectures. Furthermore, an optical neural network using photons as the medium can break through the limitations of a traditional electronic neural network and provide high speed and low loss advantages.

Progress In this paper, we analyze the applications of deep learning in micro-nano structure design and spectral response prediction, holographic imaging application, optical sensing and imaging technology, new photon-driven neural network, and other directions in detail through examples. We also list the challenges existing in the combination of deep learning and optics, and the development directions of this field have prospected. First, predicting spectral response based on existing micro-nano structures is an important step in micro-nano optics. The neural network can predict the near-field electromagnetic response, far-field scattering mode, Poynting vector, and other physical quantities without time-consuming calculations, which indicates that data-driven deep learning can simulate the propagation and distribution of electromagnetic fields ( Fig. 1). The use of neural networks can significantly improve prediction speed by several orders of magnitude. The process of designing micro-nano structures to get the ideal resonance response is called inverse design. Most early applications are designed for fixed micro-nano structures with some structural parameters as variables ( Fig. 2). There are non-unique solutions in the inverse design of micro-nano structures, which cause that the neural network is hard to converge. To solve the problem, a cascaded neural network is adopted, and deep learning is used in other complex response predictions for micro-nano structure ( Fig. 3). In addition, a generative adversarial network (GAN) can be used to improve the generalization and diversity of inverse design ( Fig. 4), but its shortcoming is poor interpretability. GAN can be upgraded to a multi-layer GAN system in the following work, which can predict more structural parameters and have various functions ( Fig. 5). Optimization algorithms commonly used in the inverse design include gradient optimization algorithms, genetic algorithms, etc. As a powerful optimization method, deep learning combined with topology is a good development prospect in micro-nano inverse design ( Fig. 7).

Second, deep learning can learn rules or detailed features from data. The advantages of deep learning can replace conventional optimization algorithms to improve image quality and work efficiently, especially in holographic imaging. Deep learning is widely used in phase recovery and image reconstruction of holograms (Fig. 8). Compared with the conventional way of generating computer-generated hologram (CGH), deep learning can improve the quality and computing speed of CGH, and ultra-lightweight real-time 3D holography can be implemented (Fig. 9).

Moreover, optical sensing and computational imaging techniques based on deep learning are a hot topic. Computational imaging technique such as single-pixel imaging based on traditional algorithms often requires scanning acquisition and calculation, which consumes a lot of computing time and power. In the case of a low sampling rate, its imaging quality cannot be guaranteed. With the advantages of data-driven deep learning, the neural network adds new vitality to single-pixel imaging technology (Figs. 11--13). Unlike computational imaging, optical sensing imaging technology employs laser-active lighting to collect optical signals via optical sensors or receivers and reproduce object information via target signal processing. Combining deep learning with non-visual imaging and replacing a single optical model with a data-driven method frequently results in high speed and adaptability, as illustrated (Figs. 15--16). Deep learning also has good applications in other optical sensing imaging fields, such as microscopic imaging, three-dimensional imaging, fusion imaging, etc.

As a kind of electromagnetic wave, light is different from an electrical signal in the propagation process and is not interfered by an external electromagnetic field. Meanwhile, as a carrier of information transmission, light has high bandwidth, high transmission speed, and low loss. Researchers began to consider using light to replace electrons to establish a neural network in the optical field. A diffractive all-optical neural network can be used in image classification, optical logic operation, spectral aiming to classification, and other tasks (Figs. 18--23). Deep learning has also been applied to other optical fields, such as optical cloaking, optical anti-counterfeiting, multichannel modeling, etc.

Conclusions and Prospects In conclusion, there are still some challenges in the combination of optics and deep learning algorithms. Here we put forward several prospects. First, physical models are incorporated into the deep learning computing process to increase the interpretability of the models to reduce the network’s dependence on data. Second, the combination of experimental and simulation data is necessary. Expanding the training set by using multiple algorithms to generate new data from real data can reduce the difficulty of training. Third, the combination of deep learning and traditional optimization algorithms can complement each other to a certain extent, improving the performance of the network model and greatly shorten the computing time. It is also beneficial to enhance the generalization and robustness of the network. Finally, an optical diffractive neural network breaks through the limitations of traditional electronic neural networks. But currently, most of the all-optical neural network is only for a single task, and it has low stability on-chip photonic neural network and other problems. More attempts and studies are needed, and there is still plenty of room for improvement.