To address the problems of detail loss, inconspicuous targets, and low contrast in infrared and visible image fusion, a fusion method combining fast joint bilateral filtering (FJBF) and an improved pulse-coupled neural network (PCNN) was proposed. The operational efficiency can be effectively improved by ensuring the quality of the fused image. First, the source images were decomposed by fast joint bilateral filtering. Second, to extract significant structure and target information, a weighted average fusion rule based on a visual saliency graph (VSM) was adopted for the basic layer image, and an improved pulse-coupled neural network model was adopted for the detail layer image. All parameters of the PCNN can be adjusted according to the input bands, and the fusion image was reconstructed using the superimposed fusion map of the base layer and the fusion map of the detail layer. The experimental results show that this method can significantly improve the image fusion effect and effectively retain important information, such as targets, background details, and edges.