We propose an efficient quantum memory scheme with controlled reversible inhomogeneous broadening in an optical cavity. When the absorption per cavity round trip exactly matches the transmission of the coupling mirror (impedance matching condition), we show that the input light field can be completely absorbed by an inhomogeneous broadening atomic ensemble. From calculation we obtain that either a forward or a backward retrieval process the quantum memory efficiency can reach unity even for a small optical depth of the atomic system. If there is no cavity, 100% efficiency is obtained only for extremely large optical depth. A high efficiency for the quantum memory at a small optical depth is easily achieved in practice. Therefore our proposal offers promising possibilities for the practical realization of high quantum memory efficiency.