To meet the power-handling capacity and compactness needs of high-power microwave system, a novel waveguide diplexer is proposed. Overmoded waveguides are introduced to improve power-handling capacity. The resonant cavity with triangular metal insert structure is introduced to design waveguide filters and waveguide bends are introduced to achieve compactness. The theoretical analysis of filter design is carried out by using microwave network method and two X-band filters are designed. Furthermore, the diplexer is formed after the size of T-junction is determined by using terminal short-circuit method. The results of simulation and practical test show that power-handling capacity of each channel of the diplexer is greater than 0.11 GW and 0.12 GW respectively, the transmission efficiency is higher than 83.9% and 82.4% respectively, and the isolation is above 20 dB. Besides, not only can the filter order be increased according to the demand, but also more waveguide bends can be introduced to improve the space utilization.