In order to improve the fusion efficiency and solve the problem of wrong selection of fusion coefficients in the fusion method based on multi-scale transform, a method based on non-subsampled dual-tree complex contourlet transform combined with adaptive block is proposed for infrared and visible image fusion. Firstly, the source image is decomposed into low-frequency and high-frequency parts by non-subsampled dual-tree complex contourlet transform. For the low frequency coefficients, an adaptive block-based fusion technique is applied, where the optimal block size can be calculated by using the improved drosophila algorithm, and the low-frequency fusion results are refined to obtain a label map which can accurately indicate the origin information of each pixel. Then, the neighborhood coefficient difference of the high-frequency component is used to combine the label map height. Finally, the fused image is reconstructed. The experimental results demonstrate that the proposed method can accelerate the process of fusion and solve the problem of block effect in spatial block fusion.