In order to overcome the problem that the global relative brightness feature of the image is ignored when the current visible and infrared image fusion methods fuse the image through the energy feature of the image, resulting in the weak expression ability of the infrared target in the fused image, a fusion method by measuring the relative brightness of the image based on the Nonsubsampled shearlet transform is designed. Firstly, the low-frequency and high-frequency coefficients of the image are analyzed by NSST transform. Then, the region energy function is used to measure the energy features contained in the region image. Based on the global mean value and regional mean value of the image, the relative brightness measure model is constructed to obtain the relative brightness feature of the regional image. The energy feature and relative brightness feature of regional image are combined to fuse the low frequency coefficients. Using the frequency values of the four dimensions of row, column and diagonal of the image, a four-dimensional detail measurement factor is established, which is used to calculate the fusion high-frequency coefficients of the image detail features, and then to obtain the fusion image. Experimental results show that the proposed algorithm can not only display the details of the image better, but also better express the content of the infrared target in the image.