In order to solve nonnegligible problems in brain tumor magnetic resonance imaging (MRI) segmentation, such as few samples, class imbalance and low accuracy of small districts, this essay proposes a new multi-scale and multitask deep-learning algorithm called TDDU-Net based on 3D No-New U-Net. Firstly, this paper applies the structure with an encoder and three different decoders to the network. Next, the ConvXt module is a pre-processor of the original decoder with a reverse bottleneck structure in order to overcome the underutilization of high-level semantics when some core regions decode. Then, InConvXt is a generalized feature processing module at the bottom layer between the encoder and decoder to ensure the accuracy of the generalized features and enhance the stability of the network. Finally, the deepwise convolution is used to reduce the calculation amount of the network parameter at the appropriate location while ensuring accuracy. The experiments show that the Dice similarity coefficients (DSCs) of the predicted segmentation in the BraTS18 dataset reaching 0.907, 0.847, 0.807 in the whole tumor region (WT), the tumor core region (TC) and the enhancing tumor region (ET). The method performs better, which is helpful in segmenting the smaller tumor area in MRI.