Inspired by the previous works on the conditional quantum logic gate, a scheme for realizing the C-NOT gate, Toffoli gate and Fredkin gate in cavity QED is proposed. The scheme only involves large-detuning atom-field interaction, so it is very simple and easy to be realized because the cavity is only virtually excited, and transfer of the quantum information between the atoms and cavity will not occur. Thus, it is insensitive to the cavity decay and the thermal field, and the requirement on quality factor of the cavities is greatly loosened and the scheme is feasible with present experimental technology.