Breast cancer is one of the malignant tumors with the highest mortality among women globally and its early detection helps to increase the survival rate of patients. In this paper, we mainly used the target detection network in deep learning to locate and classify tumor lesion areas in the X-ray mammography images. Then, the Mask R-CNN network was taken as the target detection model for the improvement of its benchmark network D-ShuffleNet. Furthermore, a new network Mask R-CNN-II was proposed, to which the transfer learning algorithm was applied. Finally, it was experimentally demonstrated that the Mask R-CNN-II network had higher detection accuracy than the Mask R-CNN network. Besides, we also found that the proposed benchmark network, the idea of image fusion applied, and the transfer learning algorithm were effective. In conclusion, the network proposed in this paper is beneficial to improve the localization and classification of breast tumors and can provide auxiliary diagnostic advice for radiologists, which has certain clinical application value.