Surface plasmon-photon hybrid waveguides have been widely used in micro-nano photonics due to their strong enhancement effect in the local field and relatively long transmission distance. Based on the finite element method, the enhancement of resonance energy transfer (RET) between two two-level atoms in a hybrid waveguide with different sizes is studied systematically. The hybrid waveguide is composed of a metal surface plasmon waveguide and a rectangular medium waveguide, with a low refractive index gap between them. The waveguide mode is solved by the two-dimensional (2D) finite element method to obtain the approximate photon dyadic Green's function in the waveguide, and it is compared with the strict three-dimensional (3D) finite element solution. The comparison results prove that for the single mode, the photon dyadic Green's function constructed by the waveguide mode agrees well with the strict solution. In addition, compared with the 3D finite element method, the 2D finite element method needs much fewer computation resources. With the help of the 2D finite element method, the size required for the single mode transmission at 1550 nm and its effect on the RET enhancement are systematically studied. It is found that a narrower waveguide is accompanied by a higher medium waveguide needed for the single mode transmission. As the waveguide widens, for a higher medium waveguide, its transmission distance increases, and the RET enhancement factor decreases. While for a lower medium waveguide, its transmission distance firstly increases and then decreases, and the RET enhancement factor declines sharply. As the height of the medium waveguide increases, the transmission distance decreases sharply at first and then increases slowly, while the RET enhancement factor increases sharply at first and then changes slowly. Furthermore, the small low refractive index gap is followed by a short transmission distance and a large RET enhancement factor.