A general analysis of the influence of two- photon or three- photon absorption on terahertz (THz) generation via optical- difference frequency generation from the nonlinear optical material is presented. We calculate the best length of crystal and the corresponding THz quantum conversion efficiency for different values of the pump peak power under phase-mismatched condition and compare the results with that in phasematched condition. From the results we know that multi- photon absorption influence difference frequency generation slightly at low pump power and the impact will be strong with the increase of pump power. Because of the effect of two- photon or three- photon absorption, difference frequency generation at high pump power requires longer crystal to achieve the maximum photon conversion efficiency for both phase- matched and phase-mismatched situations. The maximum photon conversion decreases when pump power increases.We also present approaches to reduce this adverse effect.