The transmission characteristics and transmission spectrum of a five-layer metal waveguide structure were studied under the excitation of a Gaussian beam. Firstly, the eigen equation of the waveguide were numerically solved to obtain the types and field distributions of the modes supported by the waveguide. Furthermore, assuming that the waveguide was excited by a Gaussian beam, the amplitude expressions of the different modes were calculated according to the angular spectrum distribution of the Gaussian beam and the mode coupling efficiency, and the transmission spectrum was obtained through the average power flow density at the outlet. It is found that when the refractive index of the central layer is increased by every 0.02, the minimum value of transmission spectrum shifts 35 nm each time, indicating that the waveguide can be used in a sensor for detecting central medium. The research will provide a theoretical reference for the application of waveguides in sensors and dielectric detectors.