The design of a real-time processing system for double channel image fusion is proposed. The structure and function of the fusion processing system are introduced. The designs of software and hardware in the system are described respectively. In hardware design, power supply, acquisition module and processing module design are described in detail. In software design, field-programmable gate array (FPGA) and data signal processor (DSP) design are analyzed in detail. The processing system is tested finally to show that the design can meet the requirements of real time processing. At the same time, the system can be expanded to realize multichannel image fusion processing.