Laser & Optoelectronics Progress, Volume. 60, Issue 2, 0200002(2023)
Review of Image Inpainting Methods
Figures & Tables(13)
Table 1. Summary of traditional methods on image inpainting
View tableTable 1. Summary of traditional methods on image inpainting
Category Method Year/Source Contribution Exemplar-based texture synthesis Efros et al［ 8 ］1999/ICCV Nonparametric texture synthesis Wei et al［ 9 ］2000/ACM Gaussian pyramid model based on Markov Random Field Efros et al［ 10 ］2001/ACM Simple and efficient image quilting technology Ballester et al［ 14 ］2001/IEEE Trans Matching local features by calculating image gradients Drori et al［ 11 ］2003/ACM Computing confidence for each patch Hays et al［ 12 ］2007/ACM Searching patchs within external databases He et al［ 15 ］2014/IEEE Trans Matching local features by using the statistics of similar patches Exemplar-based structure synthesis Bertalmio et al［ 17 ］2000/ACM Using Partial Differential Equation to generate linear structural patches Criminisi et al［ 18 ］2004/IEEE Texture and structure information can be transmitted simultaneously Chen et al［ 20 ］2020/Laser & Optoelectronics Progress Improved the priority calculation formula［ 18 ］with the method of refining data itemsWang［ 21 ］2020/Laser & Optoelectronics Progress Optimized the priority calculation formula［ 18 ］by introducing the local color varianceChen et al［ 22 ］2020/Laser & Optoelectronics Progress Optimized the priority calculation formula［ 18 ］by introducing the information entropy of measuring the complexity of the pixel blockCheng et al［ 19 ］2005/IEEE Optimized the priority function in Criminisi［ 18 ］Kwatra et al［ 26 ］2005/ACM TOG Using planar exemplar guidance Simakov et al［ 23 ］2008/IEEE A mathematical model for local restoration of untextured images Barnes et al［ 24 ］2009/ACM Fast stochastic calculation based on NNF Ružić et al［ 27 ］2014/IEEE Global repair algorithm combined with Markov Random Field Huang et al［ 25 ］2014/ACM TOG Using planar structure guidance Diffusion-based techniques Bertalmio et al［ 17 ］2000/ACM Diffusion method based on isophote lines Shen et al［ 28 ］2002/SIAM Combined total variation denoising model with Partial Differential Equation Telea et al［ 29 ］2004/JGT Fast Marching Method
Table 2. Summary of image inpainting methods based on deep learning
View tableTable 2. Summary of image inpainting methods based on deep learning
Category Method Year/Source Contribution Pixel-generation-based techniques Richard et al［ 35 ］2001/VIIP Fast image restoration method based on diffusion convolution kernel（Gaussian） Hadhoud et al［ 36 ］2008/SIP The position of zero weight value of diffusion convolution kernel［ 35 ］Jain et al［ 37 ］2008/NIPS A neural network structure for denoising Auto-encoder-based techniques Xie et al［ 39 ］2012/NIPS Stacked sparse denoising Auto-encoders Pathak et al［ 40 ］2016/CVPR Context encoder to capture more semantic information Iizuka et al［ 41 ］2017/ACM Global and local context discriminators added to the auto-encoder Yu et al［ 44 ］2018/CVPR A parallel encoder model based on attention mechanism Sagong et al［ 45 ］2019/CVPR A shared encoding network with two parallel decoding tasks Shin et al［ 46 ］2020/NNLS Adaptive dilated convolutional layers added to PEPSI［ 45 ］ modelYang et al［ 47 ］2017/CVPR Multi-scale neural patch synthesis approach Yan et al［ 49 ］2018/ECCV A special shift-connection layer Shift-Net Liu et al［ 51 ］2018/ECCV A partial convolution structure based on U-Net structure Xie et al［ 53 ］2019/ICCV A learnable bidirectional attention module which can automatically update the mask Liu et al［ 54 ］2019/ICCV A network architecture based on coherent attention mechanism layer Nazeri et al［ 56 ］2019/Arxiv A two-stage adversarial model EdgeConnect Li et al［ 57 ］2019/ICCV A progressive reconstruction of visual structure network Ren et al［ 58 ］2019/ICCV A novel two-stage network which can generate texture structures consistent with context semantics Zeng et al［ 62 ］2019/CVPR A pyramid context encoder network combining high-level semantics and texture information Yi et al［ 63 ］2020/CVPR A context residual aggregation network for high resolution image inpainting Li et al［ 64 ］2020/CVPR A cyclic feature inference network for recovering the large missing regions of damaged images GAN-based techniques Radford et al［ 65 ］2015/Arxiv DCGANs combining Convolutional Neural Network（CNN）and unsupervised learning Isola et al［ 55 ］2017/CVPR PatchGAN based on patch identification Yeh et al［ 66 ］2017/CVPR DGMs to repair irregular regions and capture richer semantics Lou et al［ 68 ］2018/PRRS RMSProp optimization algorithm is added to WGAN to maintain good performance on non-convex problems Yu et al［ 69 ］2019/ICCV SN-Patch GAN network based on gated convolution to improve the details and semantic accuracy of repaired results Wang et al［ 70 ］2021/IEEE The validity transfer convolution and region compound normalization modules to realize the dynamic selection of valid information Zheng et al［ 72 ］2019/CVPR PIC NET to generate a variety of repair results Zhao et al［ 73 ］2020/CVPR An unsupervised cross-space translation generative adversarial network
Table 3. Quantitative evaluation results of algorithms on common datasets
View tableTable 3. Quantitative evaluation results of algorithms on common datasets
Dataset Method PSNR SSIM FID MAE MSE Size of image Mask type（image-to-mask ratio） CelebA-HQ Sagong et al［ 45 ］25.60 0.90 256×256 Square（25%） 28.60 0.92 256×256 Irregular Shin et al［ 46 ］25.50 0.89 256×256 Square（25%） 28.50 0.92 256×256 Irregular Liu et al［ 54 ］34.69 0.98 0.72 0.04 256×256 Irregular（10%-20%） 32.58 0.98 0.94 0.07 256×256 Irregular（20%-30%） 25.32 0.92 2.18 0.37 256×256 Irregular（30%-40%） 24.14 0.88 2.85 0.44 256×256 Irregular（40%-50%） 26.54 0.93 1.83 0.27 256×256 Square（25%） Li et al［ 57 ］33.56 0.98 0.007 256×256 Irregular（10%-20%） 27.76 0.93 0.02 256×256 Irregular（30%-40%） 22.88 0.81 0.047 256×256 Irregular（50%-60%） Zhao et al［ 73 ］26.38 0.88 1.51 256×256 Irregular（10%-20%） Place2 Yu et al［ 44 ］18.91 8.60 2.10 256×256 Irregular（10%-20%） Liu et al［ 51 ］33.75 0.94 0.49 256×256 Irregular（1%-10%） 27.71 0.86 1.18 256×256 Irregular（10%-20%） 24.54 0.77 2.07 256×256 Irregular（20%-30%） 22.01 0.68 3.19 256×256 Irregular（30%-40%） 20.34 0.53 4.37 256×256 Irregular（40%-50%） 18.21 0.46 6.45 256×256 Irregular（50%-60%） Xie et al［ 39 ］25.59 0.78 1.93 256×256 Irregular（20%-30%） Liu et al［ 54 ］27.75 0.93 0.01 256×256 Square（25%） Nazeri et al［ 56 ］21.75 0.82 8.16 3.86 256×256 Irregular（25%） 24.92 0.86 4.91 2.59 256×256 Irregular（20%-30%） Li et al［ 64 ］27.75 0.93 0.014 256×256 Irregular（10%-20%） 22.63 0.81 0.038 256×256 Irregular（30%-40%） 18.92 0.59 0.076 256×256 Irregular（50%-60%） Ren et al［ 58 ］25.22 0.90 7.03 256×256 Irregular（20%-40%） Zeng et al［ 62 ］0.78 15.19 9.94 256×256 Square（25%） Yi et al［ 63 ］4.89 5.43 512×512 Irregular 4.89 5.43 1024×1024 Irregular 4.89 5.49 2048×2048 Irregular 4.89 5.50 4096×4096 Irregular Paris Street View Pathak et al［ 40 ］17.59 0.10 0.23 128×128 Square（25%） Yang et al［ 47 ］18.00 0.23 128×128 Square（25%） Yan et al［ 49 ］26.51 0.90 0.02 256×256 Irregular（10%-20%） Li et al［ 57 ］31.71 0.95 0.011 256×256 Irregular（10%-20%） 26.44 0.86 0.027 256×256 Irregular（30%-40%） 22.40 0.68 0.054 256×256 Irregular（50%-60%）
Tools
Get Citation
Copy Citation Text
Xuetao Li, Yaoxiong Wang, Fang Gao. Review of Image Inpainting Methods[J]. Laser & Optoelectronics Progress, 2023, 60(2): 0200002
Paper Information
Category: Reviews
Received: Oct. 8, 2021
Accepted: Nov. 11, 2021
Published Online: Jan. 3, 2023
The Author Email: Fang Gao (fgao@gxu.edu.cn)
© Copyright 2018-2021 | Chinese Laser Press.
All Rights Reserved 沪ICP备15018463号-20