Modern long-distance optical communication systems can significantly benefit from the MDM-WDM system, which combines mode division multiplexing (MDM) and wavelength division multiplexing (WDM) techniques to substantially increase transmission capacity and optimize spectral efficiency. However, signal transmission through passive optical fibers inevitably results in power attenuation due to inherent transmission losses, necessitating the use of optical amplifiers to restore signal power. The few-mode erbium-doped fiber amplifier (FM-EDFA) has become a key technology for improving the performance of MDM-WDM optical communication systems, thanks to its unique amplification capabilities for long-distance transmission. Nonetheless, FM-EDFA faces two major technical challenges that limit its performance and practical applications: mode-dependent gain equalization and wavelength-dependent gain flattening. In FM-EDFA, differential modal gain (DMG) causes power imbalance between different modes, directly degrading transmission quality. Furthermore, differential wavelength gain (DWG) affects the signal power distribution across wavelengths. If not addressed, these issues lead to signal distortion, ultimately compromising system stability and transmission performance.

A model for the FM-EDFA system is developed in MATLAB based on theoretical studies of erbium ion energy levels and optical power propagation equations. This model is first used to study the influence of a single-layer erbium ion doping structure on wavelength gain flattening. The effects of LP₀₁ and LP₁₁ pumps on FM-EDFA wavelength gain flattening are analyzed, with LP₁₁ pumping identified as the optimal solution. Subsequently, the single-layer erbium ion doping design is optimized to achieve wavelength gain flattening for three signal modes within the C-band. However, despite the successful wavelength gain flattening achieved with LP₁₁ pumping, the modal gain difference is too large to achieve modal gain equalization. To simultaneously achieve wavelength gain flattening and modal gain equalization, a dual-layer erbium ion doping structure is proposed, based on the mode field distribution in the fiber. Its feasibility is verified, and global optimization is performed using a genetic algorithm, which results in the determination of optimal doping design parameters. In addition, the effects of pump power, fiber length, and signal input power on FM-EDFA gain characteristics are further analyzed, and the tolerance to fabrication errors in FM-EDF manufacturing is also evaluated.

In the gain-flattening design, using single-layer erbium ion doping and LP₁₁ pumping, three signal modes with DWG of less than 0.2 dB are achieved in the C-band (Fig. 5). However, the excessive modal gain difference prevents modal gain equalization. To resolve this, a dual-layer erbium ion doping structure is designed based on the mode field distribution [Fig. 3(b)] and combined with LP₁₁ pumping to simultaneously achieve both modal gain equalization and wavelength gain flattening. Its feasibility is initially verified (Fig. 6), and global optimization using a genetic algorithm leads to the optimal FM-EDFA parameters (Table 1). The optimized FM-EDFA achieves gains above 20 dB for three signal modes in the C-band, with DMG below 0.14 dB and DWG below 0.25 dB (Fig. 7), demonstrating effective modal gain equalization and wavelength gain flattening. A parameter fluctuation analysis of FM-EDFA in practical applications shows that with pump power ranging from 300 mW to 700 mW, mode gain remains above 20 dB, while both DMG and DWG are less than 1 dB (Fig. 9). When the fiber length is varied between 3.8 m and 7.5 m, DMG remains below 0.5 dB and DWG below 1 dB (Fig. 8). When the signal input power exceeds -20 dB, both DWG and DMG stay below 1 dB (Fig. 10). Tolerance analysis of manufacturing errors in the designed FM-EDF shows that DWG and DMG could be maintained below 2 dB (Figs. 11 and 12), confirming stable performance in both modal gain equalization and wavelength gain flattening.

In this paper, we present a dual-layer erbium ion doping structure for FM-EDFA to achieve both wavelength gain flattening and modal gain equalization in MDM-WDM optical communication systems. In the analysis of the single-layer doping structure, LP₁₁ pumping and optimized doping parameters successfully achieve wavelength gain flattening but fail to ensure modal gain equalization. To address this, a dual-layer erbium ion doping structure is designed based on the mode field distribution analysis. Global optimization using genetic algorithms is employed to determine the optimal doping parameters, leading to both wavelength gain flattening and modal gain equalization. Computational results demonstrate that the optimized FM-EDFA achieves a gain above 20 dB for three signal modes in the C-band, with DMG below 0.14 dB and DWG below 0.25 dB. Furthermore, in practical applications, variations in fiber length, pump power, and signal input power have minimal influence on FM-EDFA gain performance. Tolerance analysis confirms the high robustness of the design, with the FM-EDFA exhibiting excellent stability and feasibility.