In order to investigate the influence of the photodarkening effect of ytterbium-doped fiber (YDF) on high-power and high-brightness fiber lasers, a fiber laser oscillator is established by using 25/400 μm large-mode-area YDF and 915 nm laser diodes. The maximum output power is up to 5 kW, and the output laser is near single mode. When the fiber laser oscillator is operated at full power, the photodarkening effect appears. The output power of the oscillator drops sharply, and strong transverse mode instability (TMI) occurs in the time domain of the output laser. After many tests, it is found that the pump power and output power thresholds corresponding to the TMI effect of the oscillator under the backward pump has a certain decline (about 14%), which is consistent with the traditional theory. However, the pump power and output power thresholds corresponding to the TMI effect of the oscillator under the forward pump shows a certain increase (about 15%), which is inconsistent with the traditional theory. Therefore, further investigation is required. Finally, the output power of the fiber laser oscillator fails to be maintained at 5 kW due to the photodarkening effect.