In order to fuse infrared and low light image effectively, under the premise of obtaining good fusion visual effect and achieving entropy increase, while minimizing the complexity of the algorithm, an adaptive weight fusion method is proposed based on detail enhancement. According to the characteristics of infrared and low light image, the adaptive weight matrix is calculated, and the fusion is performed for the first time, and then combined with the detail image for the second fusion. After using three conventional algorithms and the algorithm of this paper to simulate and analyze five types of typical scenes in MATLAB, and combining with subjective and objective evaluation mechanism, the algorithm in this paper has a better standard deviation and entropy increase, and the subjective fusion effect is also relatively better. Finally, it is verified in the embedded FPGA platform to realize the real-time fusion output of 50 Hz images.