To address the issue that existing restoration algorithms are prone to artifacts when dealing with large areas missing and inconsistent with the semantics of the original image, a second-order image restoration method based on gated convolution and attention transfer is proposed. The overall semantic consistency of the repair results is ensured by strengthening the influence of the internal semantics of the image to be repaired on the repair network. The rough repair results are then input into the improved refinement repair network, and the gated convolution and attention transfer network are used to repair the image’s internal texture details. The SimAM module is introduced as the attention mechanism in the encoding and decoding processes. Finally, the spectrum normalized Markov discriminator is used to determine authenticity while also providing the confrontation loss. The perceived loss and similarity loss of multiscale structure are considered as the reconstruction loss and then combined as the loss function. The comparative experiments with other image restoration methods show that the proposed method improves the structural similarity by 1.47% and the peak signal-to-noise ratio by 5.48%compared with the best results. The repair results of this method are more realistic and natural, and the ideal repair effect is achieved under various sizes.