In order to investigate the influence of the nonlinear process of four-wave mixing on the spectral broadening characteristics of high-power ytterbium-doped fiber laser, ytterbium-doped fiber with core and inner cladding diameters of 20 μm and 400 μm, resperctively, was used as the gain medium. Based on the nonlinear process of four wave mixing, an ytterbium-doped fiber laser spectral analysis model was established. The effects of pump power, pump wavelength, center wavelength, and injection mode number on the spectral broadening characteristics of an ytterbium-doped fiber laser were analyzed. The output power spectrum of 1080 nm laser and the curve of output laser linewidth with the above parameters were obtained. The simulation results show that in the nonlinear process of four-wave mixing, the increase of pump power leads to the broadening of the output spectral linewidth of the ytterbium-doped fiber laser; the output spectral linewidth broadening rate of different pump wavelengths is different; the broadening degree of the output laser spectrum of the long band central wavelength is similar; the spectral linewidth of ytterbium-doped fiber laser gradually broadens with the increase of the number of injection modes. The simulation results provide theoretical support for the study of ytterbium-doped fiber laser spectral broadening characteristics, and have certain reference significance for the engineering application of high-power fiber laser.