With the development of high power microwave sources towards high power, high frequency and long pulse, coaxial relativistic klystron amplifiers (RKA) have become one of the research hotspots in recent years. However, its development is limited by self-excited oscillation, etc. Therefore, a high Q-factor single-gap coaxial resonator is designed in this paper to suppress the self-oscillation caused by TEM mode leakage of the coaxial RKA. Through the theoretical analysis and simulation of TM₀₁ mode and TEM mode conversion in a single-gap coaxial resonator, it is found that the depth difference and axial dislocated values of the upper and lower slots of the coaxial resonator have a great influence on the change of its Q factors. When the depth difference and axial dislocation value of the upper and lower slots are 0.3 mm and 0 mm respectively, the Q factor of the coaxial resonator has the maximum value (18 764). This means that the conversion between the two modes in the resonator is minimal, greatly reducing the risk of self-oscillation between several cascaded coaxial resonators. When three cascaded high-Q-factor single-gap coaxial resonators are applied to the compact coaxial RKA, the output microwave power of the device is stable, the spectrum is pure, and there are no self-excited oscillations in the simulation and experiments.