There are differences in spatial information distribution among different medical imaging models, which is not conducive to the effective alignment of the depth feature space, resulting in the loss of shallow information in a specific area of the fusion image or excessive dependence on the information of a certain mode. To solve these problems, an asymmetric medical image fusion model based on Restormer and dual attention mechanism was proposed. Firstly, Restormer module is used to dig deep features of different modal images, and dual attention mechanism is introduced to extract global and local features of different modal images. Secondly, an asymmetric feature fusion strategy is designed, in which an independent feature encoder is designed for each mode and the extracted features are fused. Finally, the fused features are generated by the decoder. This model adopts two stages of training, the first stage mainly extracts global and local features from different modal images, and attempts to reconstruct the original image to calculate the loss; the second stage continues to extract deep features and generate fusion images. Compared with the seven mainstream image fusion models, the seven evaluation indicators, standard deviation, spatial frequency, visual information fidelity, spectral relevance, mutual information, average gradient, and Q index used to evaluate hybrid fusion have an average increase of 12.63%, 28.30%, 31.37%, 27.40%, 19.01%, 37.36%, 32.44%, respectively. The fusion strategy of this model can not only efficiently integrate the coding features from different modes, but also complete the integration of complementary information and the interaction of global information without manually designing fusion rules, and can better integrate images from different modes.