Bound states in the continuum (BICs) were initially proposed nearly a century ago by Neumann and Wigner in a quantum mechanical system [1]. Despite residing within the continuous spectrum, these states exhibit perfect localization without any radiation, attributed to destructive interference of waves [2]. In recent years, this counterintuitive mechanism has been discovered across various domains, including acoustic, water, quantum wire, and electromagnetic wave systems [3–6]. Since 2008, BIC has been revisited in coupled optical waveguide structures [7], sparking growing interest for photonic applications. In an ideal setting, BIC can facilitate extremely compact mode confinement and infinite Q-factors, leading to significantly enhanced light–matter interactions. This unique property positions BIC as an innovative platform for localizing electromagnetic waves while reducing losses in photonic structures. More recently, advancements in micro/nanofabrication technology have enabled the realization of photonic BICs in various optical systems, promising notable improvements in multiple crucial areas, such as ultralow threshold lasers, nonlinear optics, optical sensing, and topological insulators [8–13].

The straightforward approach to creating BIC involves introducing a symmetry mismatch into periodic structures, such as photonic crystals, gratings, and metasurfaces [14–17]. In these structures, BICs are realized by preventing the coupling between discrete modes and radiation modes, as they belong to symmetry-categorized eigenmodes. However, this advantage can also pose challenges, since symmetry-protected BIC makes it difficult to efficiently collect emission of these ultrahigh-Q modes due to the forbidden leaking. Therefore, recent research has focused more on quasi-BIC instead by slightly breaking the BIC condition, which can maintain a sufficiently high Q-factor while significantly enhancing emission intensity [18]. Nevertheless, the periodic structures are not always necessary. Zou et al. proposed a novel design that supports BIC even in a single waveguide. This type of BIC exists in a low-refractive-index waveguide on top of a high-refractive-index substrate, allowing for low-loss light guiding and routing [19], thus benefiting hard material platforms that are not easy to fabricate, such as diamond and lithium niobate (LiNbO3). Subsequently, this structure was experimentally demonstrated in polymer-on-LiNbO3 hybrid waveguides and microring resonators, achieving Q-factors as high as 5.8×105 [20]. Based on this architecture, various applications have been demonstrated, such as acoustic-optic modulation, second-harmonic generation, and high-dimensional optical communication [21–23].

An alternative method for generating BIC was proposed by Friedrich and Wintgen in 1985, utilizing mode coupling in the radiation channels [24,25]. Notably, compared with symmetry-protected BIC, Friedrich–Wintgen BIC offers advantages in terms of device footprint and tunability. In addition to the metasurface architecture [26], this type of BIC has also been investigated in individual dielectric Mie resonators by adjusting their structural parameters, thus demonstrating enhanced nonlinear efficiency [27–29]. Even though the modes supported inside the Mie resonator are approximately orthogonal, their far-field profiles exhibit quite large overlap, rendering interference out of the resonator possible when their frequencies are close to degeneracy. As a result, an unexpected high-Q-factor is achieved via strong coupling of pairs of leaky modes, in which the radiation loss of one mode is dramatically suppressed with destructive interference, leading to an increased Q-factor, while the other mode with constructive interference becomes more lossy. In another widely employed optical structure, i.e., the whispering-gallery-mode (WGM) microcavity, despite the extensive study of mode coupling, there are few experimental reports on the generation of Friedrich–Wintgen BIC. Traditional WGMs differ significantly from Mie resonators, as they are confined along the circular cavity boundary (i.e., microdisk and microring) through total internal reflection [30]. Consequently, the required radiation channels for external strong coupling are naturally forbidden due to geometrical rotational symmetry. As such, a critical step toward the generation of Friedrich–Wintgen BIC in WGM microcavities is the realization of controllable radiation channels that allow WGM modes to leak and then interfere in these shared continuums.

In this study, we propose and experimentally investigate a robust and general mechanism to generate Friedrich–Wintgen quasi-BIC in a single WGM microcavity. Unlike traditional circular microcavities, we introduce boundary deformation to engineer the phase space structure, enabling shared unidirectional radiation channels through the contracted unstable manifold associated with chaotic-assisted tunneling process to facilitate strong mode coupling outside the cavity. Beyond the unidirectional emissions demonstrated in various experiments [31], the advanced mode manipulation capabilities of deformed microcavities hold significant promise for several important applications, such as the suppression of laser spatiotemporal instabilities [32], novel approaches for generating experimental points [33], and phase space engineering to tailor emission channels [34], along with many others, suggesting that deformed microcavities provide greater freedom for mode control compared with traditional circular WGM microcavities. Our experimental results show avoided resonance crossing at the mode resonance frequencies, indicating strong mode coupling. Interestingly, in contrast with traditional strong coupling, the demonstrated external strong coupling exhibits a distinct characteristic in the imaginary region: the original high-Q mode exhibits an even higher Q-factor with approximately a threefold enhancement, while the low-Q mode becomes more lossy, thereby confirming the generation of Friedrich–Wintgen quasi-BIC. We present a theoretical model based on a 2×2 Hamiltonian matrix to analyze the formation of such Friedrich–Wintgen quasi-BIC and show excellent agreement with our experimental data. It is worth noting that the strong mode coupling demonstrated here not only modifies the real part, as routinely pursued in engineering mode dispersion, but also provides a flexible knob in manipulating the imaginary properties, resulting in, e.g., the demonstrated enhancement of the Q-factor. Our results hold significance for the fields of chaotic microcavities, nonlinear optics, quantum optics, and photonic integrated chips.

The realization of Friedrich–Wintgen BIC requires a shared leaking continuum where destructive interference of radiation tails occurs, as illustrated schematically in Fig. 1(a). However, as mentioned, this requirement is a challenge for conventional WGM microcavities due to their rotational symmetry. One possible solution would be directly attaching a channeling waveguide to the cavity, a technique known as “end-fire injection,” with experimental validations reported in Refs. [35–37]. Despite this approach’s achieving high output coupling efficiencies as well as high-Q-factors, identifying a suitable parameter for detuning the system to form external strong coupling remains a daunting task. We introduce deformed microcavities to fulfill this criterion due to their nature of unidirectional emissions, a property extensively studied in theory and experiments [38–41].

Among the various deformed microcavity shapes, our research focuses on the Limaçon microdisk as a proof-of-principle demonstration. The cavity boundary is defined in polar coordinates by R(ϕ)=r(1+ϵcosϕ), where r represents the cavity size, and ϵ is the deformation parameter, with ϵ=0 yielding a regular circular microdisk cavity. Figure 1(b) shows a scanning electron microscope (SEM) image of the fabricated Limaçon microdisk with ϵ=0.35, utilized in subsequent numerical simulations and experimental demonstrations. Previous studies of Limaçon microcavities have demonstrated unidirectional laser emissions and their resilience to a broad spectrum of refractive indices and deformation parameters [40], alluding to the potential for modes to externally couple via these unidirectional emission channels and generate Friedrich–Wintgen BIC. To comprehend the behavior of light propagation inside the cavity, we project the ray trajectories onto the phase space under short-wavelength approximation, known as the Poincaré surface of section. Here, we set the effective refractive index neff=1.649 to be consistent with the subsequent experimental settings. Figure 1(c) displays the corresponding 2D phase space structures for ϵ=0.35 without the coupled bus waveguide. Clearly, this regular-chaotic mixed phase space consists of three mode families: unbroken Kolmogorov–Arnold–Moser (KAM) curves near sinχ≈1 representing the quasi-WGMs; closed curves (islands) representing the regular p-bounce modes; and randomly distributed dots representing the chaotic sea. From a ray optics perspective, the quasi-WGMs [green line in Fig. 1(c)] and some p-bounce modes [red islands for six-bounce in Fig. 1(c)] can be confined in the cavity indefinitely, as they propagate above the total reflection angle (if sinχ>1/neff). However, the quantum dynamics allow a small probability for light rays to tunnel from the KAM or closed curves to the nearby chaotic sea and refractively escape the cavity quickly following Fresnel’s laws (when sinχ<1/neff).

Although chaotic rays appear randomly distributed in the phase space, their leakage follows certain channels. In Fig. 1(d), we simulate the normalized survival probability of 1500×2000 rays initially uniformly distributed in the phase space structure and derive the subsequent residual intensity distributions according to Fresnel’s law after 200 bounces. Major emissions occur at specific locations in the phase space structures, indicating that the supported high-Q modes (quasi-WGMs and p-bounce modes) share the same leaking continuum. We further verify these unidirectional emission channels using wave-optics simulations based on the eigenfrequency analysis model of COMSOL. To reduce the simulation time without compromising the generality, we calculate the normalized far-field intensity spectra of two highest-Q modes near kr=106 (k=2π/λ, λ is the wavelength, and r is the radius), as shown in Fig. 1(e). Consistent with ray-optics expectations, the high-Q modes exhibit nearly identical unidirectional emissions. Additionally, their field distributions display strong spatial overlaps, as shown in the inset figures of Fig. 1(e), where the top and middle correspond to a quasi-WGM mode and the bottom is a six-bounce mode. It is also worth noting that the simulated quasi-WGMs and six-bounce modes in Fig. 1(e) are representative examples and do not exactly correspond to Modes A and B observed in the following experimental results. In addition, besides these two mode families, other high-Q modes can also support these unidirectional emissions, as they escape following the same unstable mainfolds. Another quasi-WGM mode near kr=59 is depicted in the center of Fig. 1(e), illustrating the robustness of the constructed leaky continuum across different mode types and frequency locations. Thus, by engineering the phase space structures, the shared leaking continuum that conventional circular WGM microcavity cannot support is structured, paving the way for the creation of Friedrich–Wintgen BIC.

Considering a cavity in which two modes couple with each other, this system can be represented by a 2×2 Hamiltonian matrix [31,42] H=(E1κ1κ2E2),(1)where E1 and E2 are the energies without coupling, and κ1 and κ2 are their coupling constants. In an optical microcavity system, E is a complex value with the real part |Re[Ei]| referring to the resonant frequency and the imaginary part |Im[Ei]| determining the loss, where i∈{1,2} indexes the two states. Their Q-factors are expressed as Qi=|Re[Ei]|/2|Im[Ei]|. Near the coupling region, the eigenenergies can be derived as EA,B=E1+E22±(E1−E2)24+κ1κ2.(2)

In the internal coupling situation, the states are coupled inside the cavity. The coupling constant terms are conjugates with κ1=κ2* such that κ1κ2 is a positive real number. When 2κ1κ2<|Im(E1)−Im(E2)|, known as weak coupling, the real parts of the hybrid states will have a crossing, and their imaginary parts will be an avoided resonance crossing. As the coupling strength is weak, the hybridization of the states is insignificant. Conversely, strong coupling is achieved when 2κ1κ2>|Im(E1)−Im(E2)| and allows the real parts to be an avoided resonance crossing while causing the imaginary parts to cross. This strong coupling is more intriguing, as the coupling strength is sufficient to dramatically alter the properties of these states, thereby providing a powerful tool to manipulate the optical system for, e.g., engineering the mode dispersion to generate dark pulse soliton with high power efficiency [43,44]. Figures 2(a) and 2(b) illustrate the real and imaginary parts of the two eigenstates in the strong-coupling regime, with E1=1538.5+Δ−0.0025i, E2=1538.5+2Δ−0.005i, and κ1κ2=0.0006 as an example. Despite being extensively studied and utilized in WGM optical microcavities, internal strong coupling typically leads to increase in loss (or decrease in the Q-factor) as observed from the evolution of Mode B in Figs. 2(b) and 2(c) and many other experiments [44–46]. Most applications, however, desire high-Q modes.

The configuration for mode coupling in a shared leaking continuum, as articulated in the previous section, represents a completely distinct scenario. In contrast with coupling occurring within the cavity, the two states leak into unidirectional emission channels and couple externally to the cavity via this shared continuum. For this situation, the coupling strength terms can be κ1≠κ2*, and κ1κ2 is treated as a complex number. As depicted in Figs. 2(d)–2(f), we illustrate an example with E1=1538.5+Δ−0.023i, E2=1538.5+2Δ−0.023i, and κ1κ2=0.00084i. The parameter Δ is introduced here to adjust the energy difference and can be analogous to different mode numbers for various mode families or to resonance wavelength detuning induced by the thermo-optic effect as demonstrated in the following experimental findings. We also keep the total loss (coupling loss and intrinsic loss) constant to simplify the model. The real parts of the hybrid modes exhibit similar avoidance of resonance crossing as observed in the case of strong internal coupling [Fig. 2(a)]. Interestingly, their imaginary parts behave differently: as the detuning parameter Δ approaches zero, where their coupling is the strongest, the initially high-Q Mode A exhibits a remarkable reduction in its Im(E) and an enhancement of its Q-factor, indicating a significant suppression of loss. Conversely, Mode B, with an initially low-Q-factor, experiences a further increase in loss and a reduction in its Q-factor. Theoretically, by carefully setting the proper coupling term, the Im(E) can be close to zero, meaning the coupling to all loss channels is completely suppressed, and an infinite Q-factor can be expected. This situation is known as BIC. However, in real-world microcavity systems, not all loss channels can be completely eliminated, such as material absorption or boundary roughness-induced scattering loss. For our scenario, as marked in Fig. 2(f), the Mode A at Δ=0 indicates the formation of Friedrich–Wintgen quasi-BIC evidenced by its significant enhancement of Q-factor.

Obviously, these clear different behaviors between internal and external strong coupling scenarios highlight the crucial role of the external leaking continuum in generating Friedrich–Wintgen quasi-BICs. The shared leakage channels, as constructed in Section 2, offer strong spatial overlap for different resonant modes, thereby providing the desired complex coupling term κ1κ2. To facilitate the formation of such Friedrich–Wintgen quasi-BIC modes, effective detuning parameters are also necessary to control the resonances and achieve strong frequency overlap, including mode numbers and temperature-induced effective refractive index modulation utilized in the following of our experimental demonstrations. It is also worth noting that, although our study focuses on the strong coupling scenario, some numerical and theoretical studies have indicated that external weak coupling also significantly influences the properties of hybrid states [47,48], which also differs noticeably from conventional internal weak coupling.

We conduct the experimental demonstrations on an integrated photonic platform comprising a 335 nm stoichiometric silicon nitride (Si3N4) layer deposited by low-pressure chemical vapor deposition (LPCVD) on 4000 nm wet thermal silicon oxide on silicon wafers. The fabrication process begins with the initial patterning of 350 nm ma-N 2403 photoresist masks on the Si3N4 film using an e-beam lithography (EBL) system. This step is followed by appropriate developing and post-reflow processes to improve boundary roughness and dry etching selectivity. Subsequently, dry etching is performed using fluorine-based chemicals in an inductively coupled plasma reactive ion etching (ICP-RIE) system, followed by oxygen plasma treatment and hot Piranha cleaning to remove residual polymers. The SEM images are shown in Fig. 1(a). The etching depth of the Si3N4 layer is approximately 290 nm, in order to reduce the resonant mode numbers. Finally, the samples are manually cleaved for subsequent testing on a fiber-edge coupling setup.

To excite the resonances, we position a bus waveguide parallel to the Limaçon microdisk at ϕ=π and adjust the cavity size r, deformation parameter ϵ, waveguide width w, and waveguide-to-cavity gap g to be 120 μm, 0.35, 1.5 μm, and 700 nm, respectively. The above coupling conditions, as well as the shallow-etched depth, are selected to balance the supported number of mode families and optical loss. This dimension corresponds to neff=1.649 for transverse electric (TE) polarization modes. Only TE polarization waveguide modes are excited by carefully adjusting a fiber polarization controller before injection. The transmission spectrum is then measured using a fiber-to-waveguide edge coupling setup.

In Fig. 3, we present the experimentally recorded transmission spectrum in the vicinity of 1540 nm, revealing the excitation of three distinct sets of modes, denoted as Modes A, B, and C. These modes belong to different mode families with varying free spectral ranges (FSRs). So, at a certain region, some modes have close resonant wavelengths, thus offering strong frequency overlap to couple with each other. Mode C is ignored in the subsequent analyses, as it remains spatially isolated from Mode A and Mode B, thereby having a negligible coupling with them. From the spectrum in Fig. 3(a), it is evident that the offset between the resonances of Mode A and Mode B is clear around 1520 nm, gradually reduces as the wavelength increases, and then diverges again at longer wavelengths. Figure 3(b) is a zoom-in view on the variation in the modes’ separation at different wavelengths. Notably, the offset between the two modes is correlated with their linewidths. At a larger offset around 1520.50 nm, Mode A and Mode B exhibit comparable linewidths. However, as they approach each other, the linewidth of Mode A displays a noticeable reduction near 1530.94 nm, reaching a minimum value at a wavelength of 1538.48 nm. This suggests a substantial decline in the loss Mode A perceives. After their separation, Mode A gradually broadens and recovers to its original linewidth, while Mode B exhibits a trend opposite to that of Mode A.

To grasp a more comprehensive understanding of the evolution of Mode A and Mode B in the avoided resonance crossing region, we characterized their resonant wavelength differences and individual Q-factors, corresponding to the real and imaginary parts of the eigenenergies. The resonant wavelengths clearly exhibit a pronounced avoided resonance crossing, as depicted in Fig. 4(a), indicating strong mode coupling. The point where the two modes are closest, approximately at a wavelength of 1538.48 nm, is labeled as Mode 0. By fitting Lorentzian curves to each resonance presented in Fig. 3(a), we determine the Q-factors of Mode A and Mode B, as shown in Fig. 4(b). More details about the resonance fittings are discussed in Appendix A and shown in Fig. 7. Remarkably, with an increasing coupling strength as the two modes approach each other, we observe a significant enhancement in the Q-factor of the originally long-lived Mode A, suggesting effective suppression of the loss channel due to strong coupling in the shared leaking continuum with Mode B. Conversely, with a decreasing coupling strength as the two modes move apart, the Q-factor of Mode A is reduced, while that of Mode B slightly increases. In contrast with Mode A, the Q-factors of Mode B show a slight decrease initially with increasing mode number, followed by a gradual increase. These experimental findings match well with the theory models [as shown in Figs. 2(d)–2(f)] and offer compelling evidence for the formation of Friedrich–Wintgen quasi-BIC. At this juncture, it is important to note that other loss channels, such as waveguide coupling loss, material absorption loss, and scattering loss, cannot be completely eliminated. Therefore, only the Friedrich–Wintgen quasi-BIC, instead of true BIC, can be physically realized.

In addition to the dramatically enhanced Q-factors, the decreased coupling efficiency is most likely caused by changes in their mode distributions induced by external strong coupling. This phenomenon has been also evidenced in some numerical demonstrations [31,42]. Visualizing the quasi-BIC modes experimentally would be highly beneficial for fully understanding the underlying physics driving the Q-factor enhancement. The recently developed direct resonance mapping technique could potentially be employed to directly measure the mode distributions near regions of external strong coupling [37]. However, this technique is beyond the scope of our current research and not directly applicable to the Si3N4 platform used in this study.

Our theory models as well as the experimental results in Fig. 3 suggest that the coupling strength affects the Q-factor of Mode A. To further explore the role of the coupling strength on Friedrich–Wintgen quasi-BIC, we utilized a thermoelectric cooler (TEC) to finely tune the refractive index of the microdisk. The quantitative experimental data in Fig. 5 further corroborate the qualitative phenomena illustrated in Fig. 3. By maximizing the coupling strength through TEC-induced refractive index modulation, the highest Q-factor of Mode A exceeds 3×105, representing an enhancement of more than three times the Q-factor before external mode coupling.

By constructing the unidirectional emission channels, the above experimental evidence has demonstrated the realization of Friedrich–Wintgen quasi-BIC in the Limaçon microdisk with ϵ=0.35. Intuitively, by tweaking the deformation parameter ϵ, we can finely tune the phase space structures and the coupling conditions of the Limaçon microdisk, thereby engineering its emission channels and the properties of the generated Friedrich–Wintgen quasi-BIC. Previous experimental results have shown that a deformation parameter ϵ=0.4 corresponds to optimal directionality [49]. To validate this hypothesis, we fabricated the Limaçon microdisk with ϵ=0.4 on the same substrate, while holding all other parameters the same as those in Fig. 3. The corresponding transmission spectrum depicted in Fig. 6(a) is expected to exhibit a similar avoided resonance crossing behavior to Fig. 3. However, dramatically different from the case of ϵ=0.35, from wavelength of 1534.1 to 1547.6 nm, the resonance dip of Mode A gradually disappears as the two modes approach, indicating the presence of complete destructive interference. Subsequently, the resonance of Mode A reappears as the two modes move apart, while the coupling efficiency of Mode A also increases. This new set of data not only reaffirms the generation of Friedrich–Wintgen quasi-BIC again but also indicates that the deformation parameter constitutes an effective means to manipulate the properties of Friedrich–Wintgen quasi-BIC.

In this study, we proposed and experimentally demonstrated a robust mechanism for generating Friedrich–Wintgen quasi-BIC in WGM optical microcavities. By deforming the cavity boundary, we constructed stable unidirectional emission channels shared by different resonant modes. Through this leaking continuum, we experimentally investigated external strong coupling, resulting in a significant enhancement in the Q-factor by more than three times. While not extensively discussed in this paper, strong mode coupling also serves as a powerful tool for manipulating optical microcavity systems. It enables mode dispersion tuning, facilitates the generation of dark-pulse solitons [43], enhances power conversion efficiency in bright solitons [50], and improves the performance of squeezed-light sources [51]. Our work will create new tools for the fields of chaotic microcavities, quantum optics, nonlinear optics, and integrated photonics.

Acknowledgment. We thank S. Zhang for helping with the C++ coding of the ray-optics simulation.