Image reconstruction is one of the key steps in the optical computational imaging. At present, image reconstruction based on deep learning mainly uses convolutional neural network, cyclic neural network and generative adversarial network. Most models are only trained through the data of a single mode, which is difficult to ensure the quality of imaging while possessing the generalization ability of different scenes. To solve this problem, a multi-modal image reconstruction model based on the Transformer (Trans-MIR) is proposed in this paper. Experimental results show that Trans-MIR can extract image features from multi-modal data to achieve high-quality image reconstruction. The structural similarity of 2D universal face speckle reconstruction was as high as 0.93 and the mean square error of 3D microtubule reconstruction was as low as 10^-4. It provides inspiration for the study of multimodal image reconstruction.