With the rapid development of modern communication technology, the transmission capacity of conventional single-mode optical fibers cannot meet growing communication needs, thus leading to the development of various multiplexing technologies. Among them, mode-division multiplexing of few-mode fibers based on space-division multiplexing has attracted much attention because of its simple structure and easy fabrication. The signal-transmission process in the few-mode fiber suffers from power loss, which must be compensated for introducing an amplifier to increase the gain of each mode signal. In addition, amplification involves amplifying the signals of different modes to different degrees, resulting in inter-mode gain differences, which greatly increases the difficulty of subsequent signal processing. Therefore, the few-mode fiber amplifier must reduce the differential modal gain (DMG) while increasing the mode gain to enhance the stability and reliability of the transmission system.

A four-mode erbium-doped fiber with a double-layer step-index structure assisted by trenches was designed based on the COMSOL platform. Based on theoretical research on the energy-level structure of erbium ions as well as the steady-state and rate equations, a few-mode fiber amplifier system is built using Matlab software. The simulated-annealing algorithm realizes the optimization of the erbium ions concentration under the three-layer-doped region to ensure the gain characteristics of the designed four-mode erbium-doped fiber amplifier. The gain and gain equalization effect of pump light with wavelengths of 980 nm and 1480 nm on the FM-EDFA were compared, and the 1480-nm pump light with better amplification characteristics was selected for subsequent analysis. The stability of the designed FM-EDFA was analyzed based on four aspects, including the simulation of the noise figure of the FM-EDFA. In addition, the gain characteristics of the FM-EDFA in the C-band are studied based on the varying wavelengths of the signal light. A 1480-nm pump light was used for amplification, the signal wavelength was set to 1550 nm, the pump-light power was adjusted, and the influence of the pump-light power on the gain characteristics was analyzed. By considering the errors produced in the actual production process, the FM-EDFA is simulated to obtain gain characteristics that are close to those of the actual production state.

When a pump light with wavelength of 980 nm is used for amplification, the average gain of the four-mode signal is 24.65 dB and DMG is 2.704 dB. When a pump light with a wavelength of 1480 nm is used for amplification, the average gain of the four-mode signal is 26.12 dB. Moreover, while satisfying the high gain, the DMG is 0.106 dB, indicating a better gain balance effect. When using pump light with a wavelength of 1480 nm, the maximum noise figure of the four-mode signals is 3.71 dB. These results demonstrate that the FM-EDFA with this structure has relatively good noise characteristics. The analysis of the gain characteristics of the signal in the C-band shows that the gain of the FM-EDFA in the entire band is relatively stable and can achieve an amplification effect of more than 25.90 dB. Moreover, the DMG does not exceed 1.3 dB, which can achieve a gain equalization effect. When the pump power was changed in the range of 0.1 W and 1 W, the maximum DMG is 1.99 dB, which is always maintained below 2 dB, proving that the FM-EDFA has a good stability. Considering the production error, the average gain is 26.08 dB, the lowest gain is greater than 25.9 dB, and the DMG is 0.34 dB.

To effectively amplify and gain equalization of the four-mode multiplexed signal, a double-layer step-index few-mode erbium-doped fiber amplifier with a trench is designed. The doping range of erbium ions is designed by analyzing the mode field, and the concentration of the three layers of erbium ions in different doping ranges is optimized using a simulated annealing algorithm. The simulation analysis shows that the 1480-nm pump light can achieve a better gain equalization effect than that of 980 nm and can shorten the length of the active optical fiber. For the pump light with 1480-nm wavelength, the average gain of the four-mode multiplexing signal is greater than 26 dB, DMG is 0.106 dB, and the maximum noise is 3.71 dB. In the C-band, all four modes can achieve amplification characteristics above 25.90 dB, and the maximum value of the DMG is 1.29 dB. Considering the influence of the pump power, the DMG is always maintained below 2 dB when the pump power was changed within 0.1 W and 1 W. The gain obtained by the tolerance analysis is above 25.9 dB, and the DMG is 0.34 dB. Thus, this study proves that the designed FM-EDFA exhibits good stability characteristics.