ObjectiveHigh-order effects in optical fibers, such as third-order dispersion and fifth-order nonlinear effects, are common influencing factors in practical optical fiber transmission, significantly impacting the transmission and characteristics of optical solitons. Through systematic mathematical modeling and analysis, this study aims to reveal the impact of these high-order effects on the dynamic behavior of vector solitons and to explore the characteristics and behaviors of vector solitons. By providing theoretical foundations and experimental guidance for the application of optical solitons in fiber transmission, this research can further expand the transmission capacity of optical communication and provide essential support for the construction of future high-capacity, long-distance communication networks.MethodBased on the coupled nonlinear Schrödinger equation (NLS), this study employs an analytical approach - the variational method, to investigate the dynamic characteristics of bright and dark vector solitons in birefringent optical fibers considering third-order dispersion and fifth-order nonlinear effects. The variational method is used to derive the evolution equations for the parameters of the two types of solitons, and the impact of third-order dispersion and fifth-order nonlinearity on soliton transmission and interaction is analyzed through graphical representations.Results and DiscussionsThis study examines the dynamic characteristics of the orthogonal polarization components of bright and dark vector solitons in birefringent optical fibers considering high-order effects such as third-order dispersion and fifth-order nonlinearity. The findings are as follows: In terms of transmission characteristics, both types of vector solitons in birefringent optical fibers exhibit properties of maintaining constant amplitude and conserving energy. Positive third-order dispersion reduces the rate of change of the center position of bright solitons with transmission distance, while negative third-order dispersion increases this rate for bright solitons and has the opposite effect on the center position of dark vector solitons. Fifth-order nonlinearity does not affect the transmission speed and center position of solitons. Regarding the interaction between soliton components, fifth-order nonlinearity has no direct relationship with the interaction between the polarization components of vector solitons. Positive third-order dispersion can either enhance or weaken the effective interaction between solitons, depending on the specific range of its values, while negative third-order dispersion always enhances the interaction between solitons. Additionally, in the absence of third-order dispersion, polarization components of vector solitons that are very close to each other consistently repel each other during transmission in birefringent optical fibers. The stronger the soliton amplitude, the stronger the internal interaction, which is detrimental to the stable transmission of vector solitons.ConclusionThe research findings indicate that third-order dispersion and fifth-order nonlinearity do not affect the amplitude of vector solitons, which remains constant during transmission. Positive third-order dispersion reduces the rate of change of the center position of bright solitons with transmission distance, whereas for dark solitons, the effect is reversed. The rate of change in phase with transmission distance is linearly related to the values of the third-order dispersion coefficient and the fifth-order nonlinearity coefficient, while fifth-order nonlinearity does not affect the transmission speed and center position of solitons. Additionally, positive third-order dispersion can either enhance or weaken the interaction between soliton components, while negative third-order dispersion always enhances their interaction. In the absence of third-order dispersion, polarization components of solitons that are very close to each other consistently repel each other during transmission, with stronger amplitudes leading to stronger interactions, which are detrimental to the stable transmission of solitons; fifth-order nonlinearity has no direct effect on the interaction between soliton components. These results contribute to a deeper understanding of the dynamic behavior of vector solitons in birefringent optical fibers and provide important references for the design and optimization of optical fiber communication systems and transmission.