Aiming at the problem of low imaging quality of infrared night vision remote sensing system, a multi-modal image fusion method with separation of the target and background, to improve the imaging quality of infrared night vision. On the one hand, the attention U-Net is adopted to segment and fuse the target region of the infrared image and the visible light image, the powerful leaning ability of U-Net is taken advantage to preserve the target information of the source images; on the other hand, the background regions of the infrared image and the visible light image are decomposed through the guided filter, different fusion strategies are used to fuse base layer information and detail layer information, so that the salient regions of the background are enhanced. Compared experiments are carried on the TNO dataset, the results show that the proposed method outperforms the other compared methods on both subjective visual evaluationand objective quantitative evaluation.