Image fusion technology is the process of extracting and integrating complementary information from a set of images, and fusing them into a single image. This process aggregates more effective information, removes redundant information, and enhances the quality of information and scene perception capabilities in the image. Among them, infrared and visible image fusion is a common branch in the field of image fusion and is widely used in the field of image processing. Infrared images can capture hidden heat source targets and have strong anti-interference capabilities. Visible images have rich scene information through reflective imaging. The fusion of the two can complement the rich detail texture information in the visible image and the highlighted target information in the infrared image, obtain a clearer and more accurate description of the scene content, which is beneficial for target recognition and tracking. However, most of the current fusion methods based on deep learning focus on feature extraction and design of loss function, and do not separate public information from modal information, and use the same feature extractor to extract features of different modes without considering the differences between different modes. Based on this, this paper proposes an infrared and visible image fusion method based on a dual-branch edge convolution fusion network. First, based on the dual-branch autoencoder, an improved dual-branch edge convolution structure is proposed, which decomposes the extracted feature information into common information and modality information, and introduces an edge convolution block in each branch to better extract deep features; then a convolutional block attention module is introduced in the fusion layer to enhance the features of different modalities separately for better fusion effect; finally, based on the characteristics of the encoding and decoding network in this paper, a loss function combining reconstruction loss and fusion loss is proposed, which better retains the information of the source image. In order to verify the effectiveness of the proposed method, 10 pairs images were randomly selected on the TNO dataset and the test set of MSRS dataset respectively to test on 6 indicators, such as MSE, SF, CC, PSNR, QABF, and MS-SSIM. Firstly, four sets of ablation experiments were designed to verify the effectiveness of the edge convolution block and the convolutional block attention module. The results show that the edge convolution block can more effectively extract the features of the image, retain more edge information, and the fusion effect of the convolutional block attention module on modality information is also significantly enhanced. In addition, the optimal parameters of the loss function are found as α=5.0,μ=1.0 by grid search method. Besides, the proposed method is compared with the mainstream infrared and visible image fusion methods, including SeAFusion, SwinFuse, etc. The results show that the proposed method retains the high-brightness targets of the infrared image and the clear background of the visible image, with a higher contrast, and has a better visual effect. To be specific, the proposed method in this paper leads other methods in the four indicators of MSE, CC, PSNR and MS-SSIM, with the best overall quality. The above experimental results prove that compared with other methods, the fusion result of the proposed method can better retain the thermal radiation information of the infrared image and the texture information of the visible image, and surpasses the existing Infrared and Visible Image Fusion methods in terms of comprehensive performance. Although the experiment was only tested on the task of Infrared and Visible Image Fusion, the method in this paper can also be extended to the fusion of more than two modalities. Future work will continue to test its performance in other multi-modal information fusion tasks, and optimize the network structure to obtain better fusion results.