In this study, we propose an infrared target modeling method based on deep learning. Further, we design a conditional double adversarial autoencoding network by combining the adversarial concepts with autoencoding. By using the trained network, the expected infrared target images can be easily generated by inputting category labels and the random variables that satisfy a certain distribution. The effectiveness of the proposed model is verified using a self-built infrared dataset. The conducted experiments prove that the generated target images exhibit considerable authenticity and diversity. Finally, the randomly generated target images as supplement the small data set can effectively improve the problem of lack of training data and improve the accuracy of the recognition algorithm in the infrared imaging system.