By analyzing the existing assessment of image fusion quality, according to the characteristic of human visual system, a novel remote sensing image fusion quality assessment was proposed, which was based on the feature and structure similarity. Firstly, the spatial feature and spectral feature were extracted from the fused image and the original panchromatic (PAN) and multispectrum (MS) images, then the structure similarity of the spatial feature and spectral feature were involved to calculate the assessment index, so as to achieve the evaluation of image quality. The experimental results and objective evaluations indicate that, the evaluation of the proposed assessment is consistent with the subjective evaluation, which shows that the proposed method is an objective and effective assessment for the remote sensing image fusion quality metrics.