The rate equations of Er-doped and Yb-Er co-doped systems pumped at 0.98 μm are presented, considering the upconversion mechanisms, such as cooperative upconversion, cross relaxation, and excited state absorption. A multi-theoretical model is founded to analyze the gain characteristic of the waveguide amplifier, by two-dimensional waveguide finite element model and propagation equations with forward and backward amplified spontaneous emission. The dependences of the gain on waveguide length, pump power and doping concentration are obtained. The new theory is used to analyze the gain performance of a practical Yb-Er co-doped Al2O3 waveguide amplifier, and the analyzing results are in accordance with experimental data.