Most exemplar-based methods find the optimal matching block by translation, which fail to deal with the images when there is rotation or scaling transformation between source and damaged region. We propose an exemplar-based inpainting algorithm with rotation and scaling transformation. Firstly, the optimal patches, with rotation or scaling transformation in those damaged region, are obtained based on the local feature vectors. Then, the energy space function is expanded to adapt the transformation of patch in data filling. Finally, the damaged image with rotation or scaling transformation can be inpainted automatically. The experimental results show that the proposed method can accurately find the optimal matching block when rotation or scaling transformation exists, and achieve better effect, higher effectiveness and robustness.