The fusion of panchromatic images (Pan) and multi-spectral images (MS) is designed to generate multi-spectral images with high spatial resolution. A fusion method based on detailed information extraction is proposed to improve the quality of the fused images. First, the high-frequency components of Pan and MS are obtained by a rolling guidance filter and margin calculation, respectively. Second, the adaptive intensity-hue-saturation (AIHS) transform is used to process the high-frequency components of MS and Pan, determined by the pixel significance, to generate the corresponding intensity component (I, intensity). Then, the difference between Pan and I is calculated to obtain the detailed image. Then, the residual image is obtained by calculating the difference between the high-frequency components of Pan and MS with a guided filter. Finally, the detailed and residual images are integrated with the original MS image using the steepest descent method to obtain the final fusion result. The experimental results demonstrate that the fused images obtained by the proposed algorithm can achieve better subjective visual effect. Simultaneously, the objective evaluation indicators are better.