A rigorous formula for the calculation of the nonlinear absorption coefficient from the laser energy transmissivity was derived. Particular attention was paid to the role of the probe pulse shape playing in the calculation. A sample of glass rod with 1.2wt% Nd2O3 doping concentration was measured with a high power Nd:phosphate glass laser system. The nonlinear absorption coefficient γ= 1.2 cm TW was obtained. Furthermore, the variation of gain and laser pulse shape in the amplifier system, caused by the nonlinear absorption, was studied. The numerical calculation shows that the effect of nonlinear absorption is important. Namely, for a large laser system capable of TW order output, the total gain in case of γ=1.2 and 2.0 cm TW Arill be 18.7% and 27% less than in case of γ=0； also the output pulse duration will increase 14% and 23% more correspondingly.