An exact formula of reflection coefficient and transmission coefficient suitable for arbitrary inhomogeneous medium waveguide is derived by defining general wave number and using the analytical transfer matrix method which is considered sub-wave phase shift. Results show that the reflection coefficient and transmission coefficient are very simple and clear. And the formula is suitable for not only arbitrary inhomogeneous medium waveguide, but also arbitrary homogeneous medium waveguide in the refractive index. It is expected that this technology has prospects in the thin-film machining processing.