Optical whispering gallery mode (WGM) microcavities have been studied for a long time due to their high quality ($Q$) factors, and a variety of applications have been developed in microcavity sensors^[1-3], low threshold lasers^[4-6], and nonlinear optics^[7-9]. For typical circular WGM microcavities with rotational symmetry, rich resonance modes and complex overlap of modes exist^[10] due to their support of different light propagation orbits and closely spaced modes. The rich modes in circular cavities are useful in studies like higher-harmonic generation and optomechanical oscillation^[6]. However, the resulting complex mode distribution disturbs mode recognition and tracking, which limits other applications of WGM microcavities such as optical narrowband filters^[11] and single-mode lasing^[12]. Therefore, it is necessary to effectively control resonance modes in microcavities.

Several methods like micro-ring resonators^[13], ultrasmall microcavities^[14], microdisks with small defects^[15], and microbottles with the optimal coupling condition^[4] have been proposed to achieve mode suppression and selection. However, these methods require higher design complexity and lower fabrication tolerance^[16-18]. Alternatively, deformed and polygonal microcavities have been proposed and demonstrated to solve the mentioned problems, including Limacon-shaped microcavities^[19], quadrupole microcavities^[20], and square and hexagonal microcavities^[21-23]. These structures show an effective ability to tailor the light propagation paths due to non-conservation of the angular momentum and chaos-based dynamic tunneling, especially in polygonal microcavities to realize self-chaotic microlasers^[24]. Furthermore, the specified light orbits in polygonal cavities can be easily controlled and excited through waveguides because the light field distribution is nonuniform along the boundary with certain emission directions^[25-29]. Consequently, the confinement of light propagation in regular orbits and the formation of expected mode distribution can be realized by designing the boundary geometry of the microcavity.

In this Letter, we propose a Reuleaux-triangle resonator (RTR) with corner-cuts, where single-mode control can be realized with proper boundary parameters. A semi-classical ray dynamics model and two-dimensional full-wave simulations are employed to demonstrate the long-lived modes in the RTR. Simulation predicts a maximum extinction ratio (ER) of about 20 dB. Furthermore, the Maxwell boundary condition (MBC) phase-matching condition of plane waves is derived to calculate the angular indices and wave vectors for resonance modes. We use digital UV lithography to fabricate an EpoCore polymer-based RTR experimentally. Then the microcavity is excited through a waveguide under the optimal coupling condition, and the transmission spectra reveal single-mode dominant resonances with a $Q$ factor of $1.1\times {10}^4$.

The Reuleaux triangle exhibits geometric curiosities consisting of three rounded arcs similar to a triangle, which is another shape of constant width similar to the circle and has been employed in several applications of robotic legs^[30], optical chaotic characteristics^[31], and antireflective surfaces^[32]. A 3D schematic diagram and the corresponding 2D section of the proposed structure are shown in Figs. 1(a) and 1(b). The three corners are cut by the pink dashed circle whose radius is $R$, and the blue solid curve is the actual boundary of the RTR with the origin arcs on behalf of corner-cuts. Point $O$ is the center of the RTR’s boundary. The half arc angle of corner-cut $\gamma$ is defined as $$\gamma =\left|\arctan \left(\frac{p^2-2}{\sqrt{-p^4+8p^2-4}}\right)\right|-\frac{\pi }{6},0<p\le 1,$$(1)which is shown in Fig. 1(b). The parameter $R=pb/\sqrt{3}$ is decided by the side length of triangle $b$ and the scale factor $p$. Thus, the RTR’s boundary is dominated by $b$ and $\gamma$ corresponding to three long circular arcs and corner-cuts. The resonator and optical waveguide are separated by the coupling gap $g$, which affects the coupling efficiency.

To investigate the mode characteristics and design boundary parameters, finite element method (FEM) simulation is utilized to analyze TE polarized modes under three different boundary parameters. Instead of the three-dimensional model, we choose the two-dimensional simulation model with the effective refractive index $n$ to simplify the boundary condition and mode equation. Maxwell’s equation can be expressed as a scalar mode equation, $[{\nabla }^2+n^2(x,y){\omega }^2/c^2]\times \psi (x,y)=0$^[33].

The commercial software COMSOL MULTIPHYSICS is chosen to calculate resonance modes, where the eigenfrequency module can obtain the complex frequency KR and the quality factor $Q=-\mathrm{Re}(\mathrm{KR})/[2\mathrm{Im}(\mathrm{KR})]$^[14] of all possible modes in RTR. Figure 2 shows the compared results of mode distribution and the complex frequency plane in the traditional circular resonator and different scale factors $k$ of the RTR with $b=120\mathrm{\mu m}$. The RTR material is EpoCore with a refractive index $n_1=1.575$ and is surrounded by air. The mode distribution in the circular resonator is shown in Fig. 2(a), with the base mode revealed in Fig. 2(e). This implies that, in addition to the base mode, dense high-order modes with similar $Q$ factors exist in the circular resonator. In our RTR, it is obvious that resonance modes with high $Q$ factors cannot exist when $p$ is close to 1, namely, there are nearly no corner-cuts to support the total internal reflection (TIR), due to strong light scattering near the sharp corners. Subsequently, $p$ is varied from 0.98 and 0.97 to 0.94 with corresponding complex frequency planes indicated in Figs. 2(b)–2(d), where the $Q$ factor and mode number ($L$) are proportional to the corner-cut lengths. A notable gap exists between long-lived^[34] (high $Q$ factor) and short-lived (low $Q$ factor) modes, which is advantageous for mode identification, and $\mathrm{\Delta }\mathrm{KR}$ represents a spacing between two dimensionless resonant frequencies [Re(KR)]. $Q$ factor enhancement is attributed to the suppression of standing waves in the microcavity^[35]. Thus, as the corner-cuts become longer, many other modes with high $Q$ factor exist in the RTR. Mode field distributions of I, II, and III shown in Figs. 2(f)–2(h) are known as the quasi-WGM modes^[23]. All resonance modes are supported by the three corner-cuts, which provide critical TIR to confine light in the resonator. Notably, mode field distribution in Fig. 2(b) is similar to mode I, revealing a lower $Q$ factor. A trade-off among the parameters of $Q$, $L$, and $p$ is considered. Therefore, the corner-cuts with extremely small lengths lead to strong light scatting near the boundary, resulting in the decrease of the $Q$ factor and the number of long-lived modes, and vice versa. The mode suppression criteria used here refer to the single-mode operation in a specified free spectral range (FSR). Compared with circular resonator results, the RTR with the appropriate boundary represents a huge advantage of mode suppression. Thus, we choose $p=0.97$ and $b=120\mathrm{\mu m}$ as the final RTR for further exploration.

The ray dynamics of the RTR is studied to reveal light orbits of resonance modes by Poincaré surface of section (PSOS), which records the azimuthal angle of the reflection points $\theta$ and the angular momentum of the light ray $\sin \chi$. In the simulation, only the upper half of the PSOS has been considered due to the time-reversal invariance of ray dynamics in the RTR. We use 150 light rays with random initial incident angles to record the first 600 reflections for each light ray. Figure 3(a) shows the phase space in the PSOS of the resonator, with the horizontal critical line $\sin \chi =1/n_1$, which reveals a mixed phase space^[36] and non-conservation angular momentum. The period-3 orbit in Fig. 3(b) crossed by the critical line shows low $Q$ factors, which is similar to traditional circular-side polygonal microcavities^[22]. Stable islands are embedded in a chaotic sea shown in Fig. 3(c). Light tunneling channels leaking from islands to the chaotic sea follow the inserted arrow scheme in Fig. 3(a). Figure 3(d) is the magnification view of the top area in Fig. 3(a), with stable period-9 islands recognized as WGM modes [Fig. 3(e)]. The low-loss light confinement is demonstrated by $\sin \chi =0.85$ around corner-cuts far above the critical line. In general, the new orbits of period-9 islands forecasted by ray dynamics are built in the designed RTR, with agreement to FEM simulation results.

To demonstrate the exact ray-wave correspondence relationship and direct light leakage pathways in phase space, the wave functions can be projected in phase space on the boundary of the system via Husimi functions^[37]. The Husimi map ${\mathcal{H}}_1^{\mathrm{inc}(\mathrm{em})}$ corresponding to the eigenmode is shown in Fig. 4(a) with the complex frequency of $\mathrm{KR}=252.71+0.008i$, which indicates equal intensities in the counterclockwise (CCW) component and clockwise (CW) component. The positions of the highest intensity of the Husimi map are located at the three corner-cuts, while the second highest intensity is situated around period-9 islands corresponding with PSOS results. The Husimi map ${\mathcal{H}}_0^{(\mathrm{em})}$ in Fig. 4(b) reveals the emitting intensity along the boundary. The directional emission of the wave field near three corner-cuts has been identified as a direct tunneling channel from stable islands to the chaotic sea through light leakage evolution paths^[38]. As a result, light rays show weak leakage at corner-cuts with strong evanescent fields. Meanwhile, the light field being well-confined in period-9 islands means the field energy does not escape.

In order to explore the transmission characteristics in the RTR, the capability of effective excitation by a coupled waveguide for the desired modes is simulated. The width of the waveguide is set as 2 µm with the coupling gap $g$ of 125 nm to effectively excite the long-lived modes in the RTR. Figure 5(a) shows the excited mode with light traveling from left to right in the waveguide. The typical transmission spectrum of the initial coupling position $\varphi =0$ at wavelengths from 1540 to 1554 nm with $\mathrm{FSR}=4.12\mathrm{nm}$ is shown in Fig. 5(b). Figure 5(c) indicates that the coupling efficiency depends on the coupling position $\varphi$, where the maximum peak marked by the red bar corresponds to the field distribution in Fig. 5(a). Besides, the inset pictures depict the field distribution of another two secondary maximum peaks distinguished by the green bar. The polar angle positions of the three peaks are in accordance with stable islands, indicating the proportional relationship among the coupling efficiency and the overlap area in phase space positions between exciting fields and stable islands. Thus, the position of $\varphi =0$ is the best coupling condition and is selected for the following study. Figure 5(d) records the different transmission depths versus a variety of coupling gaps. The transmission depth approaches a maximum value of 21.35 dB when the gap is near 30 nm at the critical coupling^[39]. All transmission spectra indicate a single-mode characteristic. Meanwhile, the RTR reveals a maximum ER of about 20 dB.

Afterward, we illustrate the phase-matching condition and propagation mechanism of the period-9 island in Fig. 3(b) based on the MBC^[40]. The assumption for the independence of wave fields on the $Z$ component is necessary to study the phase matching of the plane wave fields inside the microcavity. Furthermore, when considering the case of light confined in the microcavity, the calculation of the Fresnel phase shift is crucial in determining the wave function under the MBC, in either TM or TE polarization modes. Thus, we define the $x–y$ coordinate shown in Fig. 6(a), which is original at the geometrical center of the RTR to match the wave vector direction in CW. The initial wave vector $\stackrel{\rightharpoonup }{k_{3n+1}}$ is set on the top of the corner-cut in Fig. 6(b) with an initial incident angle $\alpha$ relative to the $x$ axis. Only a group of certain values of $\stackrel{\rightharpoonup }{k}$ and $\alpha$ can satisfy the phase matching condition. In Figs. 6(b)–6(d), the incident wave vectors and corresponding reflection components are indicated to assume the initial condition. Two assumed wave functions are obtained from the initial wave vectors with $l=3n+1$ as shown in Fig. 6(b), $$E_{3n+1}=A_{3n+1}{e}^{i{k}_y(3n+1)\sqrt{3}ka/3}{e}^{i{k}_x(3n+1)-i\omega t}.$$(2)

On the reflection boundary, $\stackrel{\rightharpoonup }{k_l}$ can be defined in $x,y$ components as $[k_x(l),(k_y(l)]=k_l(\sin \alpha ,\cos \alpha )$, with the wave number $k_{l,l+1}=2\pi n/{\lambda }_{l,l+1}$. The $A_{l,l+1}$ is the field amplitudes, and $\omega =ck$. Then, we can get the Fresnel phase shift of TIR by calculating the ratio of $E_{l+1}/E_l$. For an incident angle $\alpha$, the Fresnel phase shift formula in TE polarization is $\mathrm{\Delta }{\phi }_{\mathrm{TIR}}(3n+1)=2\arctan \frac{\sqrt{{\sin }^2\alpha -n_2^2/n_1^2}}{\cos \alpha }$^[40], where $n_1$ is the effective refractive index and $n_2$ equals 1. Thus, the phase factors on the three corner-cuts are ${\mathrm{\Delta }}_{\mathrm{TIR}}=\sum _l[\mathrm{\Delta }{\phi }_{\mathrm{TIR}}(l)+k_y(l)2\sqrt{3}\mathrm{ka}/3],l=n+1,3n+1,5n+1$. We assume this mode belongs to a fundamental mode in WGM due to the ray orbits in the period-9 island. The propagation constant ${\beta }_m$ is equal to $L_{\mathrm{eff}}{n}_1/(R\lambda )$, and $L_{\mathrm{eff}}$ is the arc length $\mathcal{L}$ of the RTR in Fig. 6(a), where $\lambda$ is the resonance wavelength. The phase shifts on one-third of long arc length $\mathcal{L}$ can be calculated by $\mathrm{\Delta }{\phi }_1=L_{\mathrm{eff}}{\beta }_m/3$. Thanks to the rotational symmetry in the RTR, the phase shifts on three different long circular arcs need to be consistent. Hence, the phase factors on the three long circular arcs are ${\mathrm{\Delta }}_{\beta }=3\mathrm{\Delta }{\phi }_1-\frac{2\sqrt{3}}{3}ka\sum _l{k}_y(l),l=n+1,3n+1,5n+1$. Total phase shifts along the boundary are ${\mathrm{\Delta }}_{\phi }={\mathrm{\Delta }}_{\mathrm{TIR}}+{\mathrm{\Delta }}_{\beta }$. Finally, the self-consistent phase matching of the field amplitude derives the necessary condition of $$3\mathrm{\Delta }{\phi }_1+\sum _l\mathrm{\Delta }{\phi }_{\mathrm{TIR}}(l)=2\pi n,n\in Z^{*}.$$(3)

Equation (3) should be satisfied to solve the angular mode numbers $L$, incident angle $\alpha$, and resonance wavelengths ${\lambda }_i$, with the $\mathrm{FSR}={\lambda }_{i+1}-{\lambda }_i$. A series of resonance wavelengths are calculated from 1527.43 to 1582.33 nm, where the parameter $n$ must be an integer chosen between 363 to 376. Figure 6(e) shows the contrast of simulation and theoretical FSR with coincident trends. The average error rate of 1.023% with 0.131 standard deviations is depicted in the inset of Fig. 6(e), and the error rate data fall within the $1.5\times$ interquartile range (IQR), indicating low variability and high accuracy of theoretical results. The corresponding incident angle $\alpha$ is around 60°, which is in accord with the PSOS analysis results of 58.2°.

To verify the RTR’s simulation transmission spectrum and mode characteristics, the polymer waveguide and RTR (EpoClad and EpoCore negative photoresists, Micro Resist Technology) combined with a UV lithography system are fabricated below. Epo polymers are the appropriate candidates for preparing the RTRs due to the high transparency and low optical loss at the communication wavelength. The fabrication process of the polymer RTR is shown in Fig. 7(a). Especially, the FR4 sample is diced on both sides to ensure direct coupling of light between the end face of waveguides and lensed fibers with low loss. The thickness and width of waveguides measured by the profilometer are both about 5 µm. The top view of the device characterized by scanning electron microscope (SEM) is shown in Fig. 7(b) according to the designed size of the RTR, with the dimension almost the same as the ideal one. The amplified image of one of the corner-cuts shown in Fig. 7(c) reveals the smooth and high verticality of sidewalls. Partial adhesion occurring near the coupling region [Fig. 7(d)] can be attributed to the fact that our digital UV lithography system has a minimum resolution of 800 nm, which may be larger than the designed coupling gap in the lithography mask. In the experiment, the coupling gap is the designed value. The adhesion region is inversely proportional to the designed coupling gap, which can be eliminated using a higher lithographic accuracy. Therefore, the designed coupling gap is used as a reference value instead of the actual gap to characterize the optical performance of the RTR.

The experimental setup is shown in Fig. 8(a), where light from the tunable semiconductor laser (TSL) at wavelengths from 1540 to 1554 nm is coupled into and out of the waveguides in TE polarization through a polarization controller (PC). Lensed single-mode fiber is used in edge-coupling^[41], where the relative coupling positions between lensed fibers and end faces of waveguides are precisely adjusted by 3-axis motion stages. Transmission spectra of RTRs are detected by an oscilloscope (OSC) through a photodetector (PD). Figures 8(b) and 8(c) show the mode distributions in the fabricated EpoCore waveguide with the size of $5\mathrm{\mu m}\mathrm{width}\times 5\mathrm{\mu m}\text{height}$, which only supports the low-loss fundamental modes of ${\mathrm{TM}}_0$ and ${\mathrm{TE}}_0$.

Figure 9(a) shows the normalized transmission spectra around 1550 nm under TE polarization at the designed coupling gap of 300 and 500 nm, respectively. Corresponding FSRs are 4.01 and 4.03 nm, consistent with the simulation result of 4.12 nm. A series of TE modes with radial order $q=1$ marked in the transmission spectra are identified by Eq. (3) in continuous angular orders. For example, the experiment ${\mathrm{TE}}_{1,x}$ mode at 1549.7 nm obtains the number of angular modes $x=330$, along with the resonance wavelengths that are similarly calculated and detected. Furthermore, WGM spectra including several small dips with low $Q$ and low effective coupling can be identified as chaotic modes^[38]. The ER of the dominant TE mode is about 4.6 dB. The magnified view of the typical modes ${\mathrm{TE}}_{1,331}$ in transmission spectra under a 500 nm coupling gap is shown in Fig. 9(b). The Lorentz fitting curve indicates the maximum experimental $Q$ factor of $1.1\times {10}^4$, close to the simulation one of $Q=1.5\times {10}^4$. Figure 9(c) shows the functions of coupling efficiency indicated by error bars and the average $Q$ factor varying with the designed coupling gap. The variation trend of the experimental coupling efficiency is in agreement with the simulation results shown in Fig. 5(d). This system is over-coupling at the designed coupling gap of 50 nm with a $Q$ factor only about $1.7\times {10}^3$. Similarly, the $Q$ factor remains stable at about $1.1\times {10}^4$, while the coupling efficiency monotonously decreases as the gap increases larger than 300 nm, indicating that the gap exceeding 300 nm may drive the system to an under-coupling region. Compared with previous reports of different deformed microcavities for mode suppression to realize double or multi-dominant modes, our structure indicates efficient single-mode operation. Eliminating the adhesion of the coupling region and fabricating smoother sidewalls can be realized through electron beam exposure instead of UV light^[42], which will further enhance the measured $Q$ factor.

In summary, we have theoretically and experimentally studied the mode and phase space distribution characteristics of the proposed novel RTR with corner-cuts. The boundary parameters are analyzed and optimized via phase space to realize the single-mode control of long-lived 9-period islands in the RTR with a maximum ER of about 20 dB. The phase matching condition under the MBC of single modes is deduced with the average error rate of only 1.023%, which shows agreement with experimental results. Experimental transmission spectra indicate dominant single modes with a $Q$ factor of 1.1 × 10⁴. With the advantages of simple structure and single-mode selection, the RTR has tremendous potential for single-mode lasers and nonlinear optics.