Existing deep learning methods only use deep layer features for recognizing cancer and ignore the spatial information stored in the output of the surface network, yielding unsatisfactory recognition accuracy. To further promote clinical applications and aid doctors improve the consistency and efficiency of breast cancer pathological diagnosis, an improved Inception-v3 image classification optimization algorithm is proposed. This algorithm optimizes the network model through model improvement and transfer learning. Breast cancer was classified based on the pathological images of a large open database. The improved model of the proposed algorithm is superior to the traditional deep learning method, with an accuracy rate of 96%, which effectively improves the performance of the deep learning model for breast cancer diagnosis. Moreover, the proposed algorithm lays a theoretical and practical foundation for further clinical applications.