We considered the residual between an ideal high spatial resolution multi-spectral image and a pansharpened image as generalized noise, and thus proposed a deep residual denoising network (DnCNN)-based quality boosting method for the pansharpened image. We used the DnCNN to learn the patterns of detail loss and spectral distortion of the fixed fusion algorithm, and mapped the input pansharpened image to a residual image. Then, we used the residual image to compensate and repair the pansharpened image. In an experiment using the QuickBird dataset, images pansharpened using different methods were enhanced via the proposed method. The experimental results demonstrate that, using the proposed method, the qualities of all pansharpened images are improved and the best boosting is attained when this method is used in conjunction with the support value transform based method. The proposed method outperforms latest methods.