Photonics provides AI not only
with the tools to sense and
communicate more effectively,
but also with the instruments
to accelerate the inference
speed. Moreover, AI offers
photonics the intelligence
to process, analyze and
interpret the sensed data,
but also to solve a wide
class of inverse problems
in photonics design,
imaging and wavefront
reconstruction in ways
not possible before.
The editorial introduces the joint theme issue of Advanced Photonics and Advanced Photonics Nexus, “Photonics and AI,” which showcases the latest research at the intersection of these two disciplines.
Photonic computing has recently become an interesting paradigm for high-speed calculation of computing processes using light–matter interactions. Here, we propose and study an electromagnetic wave-based structure with the ability to calculate the solution of partial differential equations (PDEs) in the form of the Helmholtz wave equation, ∇ 2f ( x , y ) + k2f ( x , y ) = 0, with k as the wavenumber. To do this, we make use of a network of interconnected waveguides filled with dielectric inserts. In so doing, it is shown how the proposed network can mimic the response of a network of T-circuit elements formed by two series and a parallel impedances, i.e., the waveguide network effectively behaves as a metatronic network. An in-depth theoretical analysis of the proposed metatronic structure is presented, showing how the governing equation for the currents and impedances of the metatronic network resembles that of the finite difference representation of the Helmholtz wave equation. Different studies are then discussed including the solution of PDEs for Dirichlet and open boundary value problems, demonstrating how the proposed metatronic-based structure has the ability to calculate their solutions.
Demetri Psaltis (École Polytechnique Fédérale de Lausanne) discusses advances in optical computing, in conversation with Guohai Situ (Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences), for the Advanced Photonics theme issue on AI and Photonics.
The article comments on a recently developed neural network that enables ultrathin flat optics imaging in full color.
Metal halide perovskite materials have rapidly advanced in the perovskite solar cells and light-emitting diodes due to their superior optoelectronic properties. The structure of perovskite optoelectronic devices includes the perovskite active layer, electron transport layer, and hole transport layer. This indicates that the optimization process unfolds as a complex interplay between intricate chemical crystallization processes and sophisticated physical mechanisms. Traditional research in perovskite optoelectronics has mainly depended on trial-and-error experimentation, a less efficient approach. Recently, the emergence of machine learning (ML) has drastically streamlined the optimization process. Due to its powerful data processing capabilities, ML has significant advantages in uncovering potential patterns and making predictions. More importantly, ML can reveal underlying patterns in data and elucidate complex device mechanisms, playing a pivotal role in enhancing device performance. We present the latest advancements in applying ML to perovskite optoelectronic devices, covering perovskite active layers, transport layers, interface engineering, and mechanisms. In addition, it offers a prospective outlook on future developments. We believe that the deep integration of ML will significantly expedite the comprehensive enhancement of perovskite optoelectronic device performance.
Planar cameras with high performance and wide field of view (FOV) are critical in various fields, requiring highly compact and integrated technology. Existing wide FOV metalenses show great potential for ultrathin optical components, but there is a set of tricky challenges, such as chromatic aberrations correction, central bright speckle removal, and image quality improvement of wide FOV. We design a neural meta-camera by introducing a knowledge-fused data-driven paradigm equipped with transformer-based network. Such a paradigm enables the network to sequentially assimilate the physical prior and experimental data of the metalens, and thus can effectively mitigate the aforementioned challenges. An ultra-wide FOV meta-camera, integrating an off-axis monochromatic aberration-corrected metalens with a neural CMOS image sensor without any relay lenses, is employed to demonstrate the availability. High-quality reconstructed results of color images and real scene images at different distances validate that the proposed meta-camera can achieve an ultra-wide FOV (>100 deg) and full-color images with the correction of chromatic aberration, distortion, and central bright speckle, and the contrast increase up to 13.5 times. Notably, coupled with its compact size (< 0.13 cm3), portability, and full-color imaging capacity, the neural meta-camera emerges as a compelling alternative for applications, such as micro-navigation, micro-endoscopes, and various on-chip devices.
The global chip industry is grappling with dual challenges: a profound shortage of new chips and a surge of counterfeit chips valued at $75 billion, introducing substantial risks of malfunction and unwanted surveillance. To counteract this, we propose an optical anti-counterfeiting detection method for semiconductor devices that is robust under adversarial tampering features, such as malicious package abrasions, compromised thermal treatment, and adversarial tearing. Our new deep-learning approach uses a RAPTOR (residual, attention-based processing of tampered optical response) discriminator, showing the capability of identifying adversarial tampering to an optical, physical unclonable function based on randomly patterned arrays of gold nanoparticles. Using semantic segmentation and labeled clustering, we efficiently extract the positions and radii of the gold nanoparticles in the random patterns from 1000 dark-field images in just 27 ms and verify the authenticity of each pattern using RAPTOR in 80 ms with 97.6% accuracy under difficult adversarial tampering conditions. We demonstrate that RAPTOR outperforms the state-of-the-art Hausdorff, Procrustes, and average Hausdorff distance metrics, achieving a 40.6%, 37.3%, and 6.4% total accuracy increase, respectively.
Quantitative phase imaging (QPI) is a label-free technique that provides optical path length information for transparent specimens, finding utility in biology, materials science, and engineering. Here, we present QPI of a three-dimensional (3D) stack of phase-only objects using a wavelength-multiplexed diffractive optical processor. Utilizing multiple spatially engineered diffractive layers trained through deep learning, this diffractive processor can transform the phase distributions of multiple two-dimensional objects at various axial positions into intensity patterns, each encoded at a unique wavelength channel. These wavelength-multiplexed patterns are projected onto a single field of view at the output plane of the diffractive processor, enabling the capture of quantitative phase distributions of input objects located at different axial planes using an intensity-only image sensor. Based on numerical simulations, we show that our diffractive processor could simultaneously achieve all-optical QPI across several distinct axial planes at the input by scanning the illumination wavelength. A proof-of-concept experiment with a 3D-fabricated diffractive processor further validates our approach, showcasing successful imaging of two distinct phase objects at different axial positions by scanning the illumination wavelength in the terahertz spectrum. Diffractive network-based multiplane QPI designs can open up new avenues for compact on-chip phase imaging and sensing devices.
Optical superoscillation enables far-field superresolution imaging beyond diffraction limits. However, existing superoscillatory lenses for spatial superresolution imaging systems still confront critical performance limitations due to the lack of advanced design methods and limited design degree of freedom. Here, we propose an optical superoscillatory diffractive neural network (SODNN) that achieves spatial superresolution for imaging beyond the diffraction limit with superior optical performance. SODNN is constructed by utilizing diffractive layers for optical interconnections and imaging samples or biological sensors for nonlinearity. This modulates the incident optical field to create optical superoscillation effects in three-dimensional (3D) space and generate the superresolved focal spots. By optimizing diffractive layers with 3D optical field constraints under an incident wavelength size of λ, we achieved a superoscillatory optical spot and needle with a full width at half-maximum of 0.407λ at the far-field distance over 400λ without sidelobes over the field of view and with a long depth of field over 10λ. Furthermore, the SODNN implements a multiwavelength and multifocus spot array that effectively avoids chromatic aberrations, achieving comprehensive performance improvement that surpasses the trade-off among performance indicators of conventional superoscillatory lens design methods. Our research work will inspire the development of intelligent optical instruments to facilitate the applications of imaging, sensing, perception, etc.
Optical computing is considered a promising solution for the growing demand for parallel computing in various cutting-edge fields that require high integration and high-speed computational capacity. We propose an optical computation architecture called diffraction casting (DC) for flexible and scalable parallel logic operations. In DC, a diffractive neural network is designed for single instruction, multiple data (SIMD) operations. This approach allows for the alteration of logic operations simply by changing the illumination patterns. Furthermore, it eliminates the need for encoding and decoding of the input and output, respectively, by introducing a buffer around the input area, facilitating end-to-end all-optical computing. We numerically demonstrate DC by performing all 16 logic operations on two arbitrary 256-bit parallel binary inputs. Additionally, we showcase several distinctive attributes inherent in DC, such as the benefit of cohesively designing the diffractive elements for SIMD logic operations that assure high scalability and high integration capability. Our study offers a design architecture for optical computers and paves the way for a next-generation optical computing paradigm.
Machine-learning techniques have gained popularity in nanophotonics research, being applied to predict optical properties, and inversely design structures. However, one limitation is the cost of acquiring training data, as complex structures require time-consuming simulations. To address this, researchers have explored using transfer learning, where pretrained networks can facilitate convergence with fewer data for related tasks, but application to more difficult tasks is still limited. In this work, a nested transfer learning approach is proposed, training models to predict structures of increasing complexity, with transfer between each model and few data used at each step. This allows modeling thin film stacks with higher optical complexity than previously reported. For the forward model, a bidirectional recurrent neural network is utilized, which excels in modeling sequential inputs. For the inverse model, a convolutional mixture density network is employed. In both cases, a relaxed choice of materials at each layer is introduced, making the approach more versatile. The final nested transfer models display high accuracy in retrieving complex arbitrary spectra and matching idealized spectra for specific application-focused cases, such as selective thermal emitters, while keeping data requirements modest. Our nested transfer learning approach represents a promising avenue for addressing data acquisition challenges.
We propose an approach for recognizing the pose and surface material of diverse objects, leveraging diffuse reflection principles and data fusion. Through theoretical analysis and the derivation of factors influencing diffuse reflection on objects, the method concentrates on and exploits surface information. To validate the feasibility of our theoretical research, the depth and active infrared intensity data obtained from a single time-of-flight camera are initially combined. Subsequently, these data undergo processing using feature extraction and lightweight machine-learning techniques. In addition, an optimization method is introduced to enhance the fitting of intensity. The experimental results not only visually showcase the effectiveness of our proposed method in accurately detecting the positions and surface materials of targets with varying sizes and spatial locations but also reveal that the vast majority of the sample data can achieve a recognition accuracy of 94.8% or higher.
Leveraging an optical system for image encryption is a promising approach to information security since one can enjoy parallel, high-speed transmission, and low-power consumption encryption features. However, most existing optical encryption systems involve a critical issue that the dimension of the ciphertexts is the same as the plaintexts, which may result in a cracking process with identical plaintext-ciphertext forms. Inspired by recent advances in computational neuromorphic imaging (CNI) and speckle correlography, a neuromorphic encryption technique is proposed and demonstrated through proof-of-principle experiments. The original images can be optically encrypted into event-stream ciphertext with a high-level information conversion form. To the best of our knowledge, the proposed method is the first implementation for event-driven optical image encryption. Due to the high level of encryption data with the CNI paradigm and the simple optical setup with a complex inverse scattering process, our solution has great potential for practical security applications. This method gives impetus to the image encryption of the visual information and paves the way for the CNI-informed applications of speckle correlography.
Two mainstream approaches for solving inverse sample reconstruction problems in programmable illumination computational microscopy rely on either deep models or physical models. Solutions based on physical models possess strong generalization capabilities while struggling with global optimization of inverse problems due to a lack of sufficient physical constraints. In contrast, deep-learning methods have strong problem-solving abilities, but their generalization ability is often questioned because of the unclear physical principles. In addition, conventional deep models are difficult to apply to some specific scenes because of the difficulty in acquiring high-quality training data and their limited capacity to generalize across different scenarios. To combine the advantages of deep models and physical models together, we propose a hybrid framework consisting of three subneural networks (two deep-learning networks and one physics-based network). We first obtain a result with rich semantic information through a light deep-learning neural network and then use it as the initial value of the physical network to make its output comply with physical process constraints. These two results are then used as the input of a fusion deep-learning neural work that utilizes the paired features between the reconstruction results of two different models to further enhance imaging quality. The proposed hybrid framework integrates the advantages of both deep models and physical models and can quickly solve the computational reconstruction inverse problem in programmable illumination computational microscopy and achieve better results. We verified the feasibility and effectiveness of the proposed hybrid framework with theoretical analysis and actual experiments on resolution targets and biological samples.
Deep learning has transformed computational imaging, but traditional pixel-based representations limit their ability to capture continuous multiscale object features. Addressing this gap, we introduce a local conditional neural field (LCNF) framework, which leverages a continuous neural representation to provide flexible object representations. LCNF’s unique capabilities are demonstrated in solving the highly ill-posed phase retrieval problem of multiplexed Fourier ptychographic microscopy. Our network, termed neural phase retrieval (NeuPh), enables continuous-domain resolution-enhanced phase reconstruction, offering scalability, robustness, accuracy, and generalizability that outperform existing methods. NeuPh integrates a local conditional neural representation and a coordinate-based training strategy. We show that NeuPh can accurately reconstruct high-resolution phase images from low-resolution intensity measurements. Furthermore, NeuPh consistently applies continuous object priors and effectively eliminates various phase artifacts, demonstrating robustness even when trained on imperfect datasets. Moreover, NeuPh improves accuracy and generalization compared with existing deep learning models. We further investigate a hybrid training strategy combining both experimental and simulated datasets, elucidating the impact of domain shift between experiment and simulation. Our work underscores the potential of the LCNF framework in solving complex large-scale inverse problems, opening up new possibilities for deep-learning-based imaging techniques.
Phase recovery, calculating the phase of a light wave from its intensity measurements, is essential for various applications, such as coherent diffraction imaging, adaptive optics, and biomedical imaging. It enables the reconstruction of an object’s refractive index distribution or topography as well as the correction of imaging system aberrations. In recent years, deep learning has been proven to be highly effective in addressing phase recovery problems. The two most direct deep learning phase recovery strategies are data-driven (DD) with supervised learning mode and physics-driven (PD) with self-supervised learning mode. DD and PD achieve the same goal in different ways yet there is a lack of necessary research to reveal similarities and differences. Therefore, we comprehensively compare these two deep learning phase recovery strategies in terms of time consumption, accuracy, generalization ability, ill-posedness adaptability, and prior capacity. What is more, we propose a co-driven strategy of combining datasets and physics for the balance of high- and low-frequency information.
We propose pattern self-referenced single-pixel common-path holography (PSSCH), which can be realized using either the digital-micromirror-device (DMD) based off-axis scheme or the DMD-based phase-shifting approach, sharing the same experimental setup, to do wavefront reconstructions. In this method, each modulation pattern is elaborately encoded to be utilized to not only sample the target wavefront but also to dynamically introduce the reference light for single-pixel common-path holographic detection. As such, it does not need to intentionally introduce a static reference light, resulting in it making full use of the pixel resolution of the modulation patterns and suppressing dynamically varying noises. Experimental demonstrations show that the proposed method can not only obtain a larger field of view than the peripheral-referenced approach but also achieve a higher imaging resolution than the checkerboard-referenced approach. The phase-shifting-based PSSCH performs better than the off-axis-based PSSCH on imaging fidelity, while the imaging speed of the latter is several times faster. Further, we demonstrate our method to do wavefront imaging of a biological sample as well as to do phase detection of a physical lens. The experimental results suggest its effectiveness in applications.