Because natural image filter response probability model is often a non-Gaussian form(sparse), it leads to the optimization problem of image restoration to be a non-convex and the optimal estimation solution can not be obtained by traditional maximum a posterior estimation. Therefore, this paper proposes a image restoration algorithm based on adaptive weight matrix to solve the problem. With image restoration, a prior model for the natural image is used to restrain the ring effect effectively and the conjugate gradient algorithm based on adaptive weight matrix is used to solve the problem of the non-convex optimization function due to the sparse prior. The weight matrix updates according to the last iteration result and is able to correct the error of the local image derivative estimation in the last iteration process. Experiments show that Peak Signal to Noise Ratio( PSNR)gotten by proposed algorithm is 36.131 6, better than that from other algorithms. Finally, the panoramic image is restored by proposed method and the good results are also obtained, which demonstrates that the algorithm proposed is practical and effective.