In order to solve the problem of unstable training and low recognition accuracy of traditional deep learning algorithm in handwritten digit recognition when training samples are insufficient, a recognition method based on conditional generative adversarial network is proposed. Firstly, on the basis of the conditional generative adversarial network, the generator of category labels is used to control image generation. The image samples generated by the generator are used to expand the training data set. At the same time, the deconvolution network and the convolutional network are used to construct the network structure of generator and discriminator respectively, and the full connection layer is removed to improve the stability of the model. Then, the conditional batch normalization is introduced. Taking advantage of its use of category labels to make the network learn more features. Finally, the discriminator is improved to be a classifier, and a novel loss function is proposed to speed up the convergence rate and improve the recognition accuracy. The experimental results show that the handwritten digit recognition method proposed in this paper has better image quality and higher recognition accuracy of 99.43%. This paper provides a reference for the application of generative adversarial network and its variants in the field of image recognition.