With the development of informatization, there is a growing demand for information transmission and processing. In this situation, it becomes urgent to increase the channel capacity. Currently, there are multiple multiplexing techniques available to enhance channel capacity, such as wavelength division multiplexing, polarization multiplexing, and mode division multiplexing (MDM). Among them, MDM utilizes different modes and polarizations of light to carry data and parallelly transmits multiple data channels in optical waveguides or fibers using only a single wavelength laser source, which can be seen as a new dimension to expand the capacity of optical fiber communication. The lithium niobate-on-insulator (LNOI) platform, with its strong electro-optic effect, low material loss, and wide transparent window, can achieve high-speed electro-optic modulators and optical nonlinear devices while providing high refractive index contrast waveguides, which thus makes it capable of manufacturing high-speed and high-density on-chip optical devices. However, there are few reports on MDM devices on LNOI platforms, and they mainly focus on the principle of phase matching of directional couplers loaded with LNOI platforms with silicon nitride. Although the mode converter based on the principle of phase matching of directional couplers has low processing difficulty and good scalability, it may have the problem of a relatively large footprint, which is not conducive to large-scale on-chip integration.

According to the coupled mode theory, when light propagates in a medium, the energy of the light field can be coupled from one mode to another mode by designing the appropriate medium structure. In this process, the perturbation of the medium structure not only satisfies the phase matching requirement of the two converting modes along the propagation direction of the z-axis but also has an appropriate refractive index distribution in the lateral direction to obtain an appropriate coupling coefficient and achieve a shorter coupling length. Metasurfaces are two-dimensional artificial materials with subwavelength features that can manipulate the phase, amplitude, and polarization of light waves at subwavelength scales through a special refractive index distribution. Integrating metasurfaces into optical waveguides can help deal with the relatively large footprint of converters based on the principle of phase matching of directional couplers, which is not conducive to large-scale on-chip integration. By leveraging the subwavelength-scale manipulation of light waves offered by metasurface structures, this study proposes a compact lithium niobate (LN) waveguide mode converter that can achieve TE₀-TE₁ or TE₀-TE₂ conversion in LN waveguides. In order to achieve coupling between modes, a reverse design method is adopted, and three-dimensional (3D) electromagnetic field simulation is utilized to optimize the subwavelength periodic stripe etching structure parameters of the device, so as to meet the phase matching requirements along the propagation direction and the lateral direction refractive index distribution with a high coupling coefficient.

Figure 3 shows the top view of the designed TE₀-TE₁ LN waveguide mode converter, the simulated modal field distribution, insertion loss, and crosstalk between modes, as well as the modal field distribution of the cross-sectional planes that are perpendicular to the propagation directions at the input and output. It can be seen that the energy of the TE₀ mode gradually decreases during propagation and is gradually converted into that of the TE₁ mode. Within the bandwidth of 1400-1700 nm, the insertion loss is less than 0.8 dB, with a minimum of 0.3 dB at 1520 nm; the crosstalk between modes is less than -10 dB, with a minimum of -38 dB at 1473 nm, and the extinction ratio is 37.3 dB. The low crosstalk between modes means that most of the energy of the TE₀ mode is converted into the energy of the TE₁ mode, and the energy loss is not significant, thus making the mode converter suitable for the field of optical communication. To demonstrate the scalability of this design, the TE₀-TE₂ mode conversion design is also presented in Fig. 4. It shows that the energy of the TE₀ mode gradually decreases during propagation and is gradually converted into that of the TE₂ mode. Within the bandwidth of 1400-1700 nm, the insertion loss is less than 2.4 dB, and the crosstalk between modes is less than -10 dB. The low crosstalk between modes means that most of the energy of the TE₀ mode is converted into the energy of the TE₂ mode, and the energy loss is not significant. In order to evaluate the effect of process errors on the performance of the designed structure and ensure the reproducibility of the device, finite-difference time-domain (FDTD) simulations of the insertion loss and crosstalk between modes in TE₀-TE₁ and TE₀-TE₂ mode conversions are carried out for processing errors of the etching groove width d and etching groove sidewall angle α (Figs. 5-8). It can be seen that the designed device has good tolerance to process errors in d and α.

This study proposes a compact LN waveguide mode converter based on metasurface structures, which can achieve TE₀-TE₁ and TE₀-TE₂ conversions. In order to achieve efficient coupling between modes, a reverse design method is adopted, and 3D electromagnetic field simulation is utilized to optimize the parameters of the tilted periodic sub-wavelength stripe etching structure of the device, which complies with the phase matching requirements of mode conversion along the propagation direction and the transverse refractive index distribution requirements of short coupling length. Simulation results show that the device has an insertion loss of less than 0.8 dB and crosstalk between modes of less than -10 dB in the wavelength range of 1400-1700 nm for TE₀-TE₁ conversion, with a conversion length of about 20 μm. In addition, the device has good scalability and process tolerance for higher-order mode conversion, making it a good candidate for mode converters in high-density integrated MDM systems in future LNOI.