By solving the coupled power equations of two-mode waveguide, the relations of power attenuation and conversion properties of guided modes to input and output powers from the optical waveguide have been derived. In the glass waveguide formed by ion exchange, holographic grating and glass prism were used as input and output couplers respectively, input and output powers from the optical waveguide were measured, and the power attenuation and conversion coefficients of guided modes have been determined. Experiments show that this measurement method, is simple, effective and specifically applicable to the measurements of short waveguide samples.