Although infrared polarization imaging systems have been developed rapidly and widely, a model for evaluating their performance has not been sufficiently developed. Performance models that can match advanced polarization imaging systems are urgently required. Regarding the similarity between the training process of a deep learning network and the process of extracting cognitive information from the human brain, this paper introduces a deep learning method in the field of system performance modeling for the first time and proposes a performance model for infrared polarization imaging systems that can automatically evaluate system performance based on two-dimensional images. The model includes two main modules: a degradation module and a performance awareness module. When evaluating a new system, high-quality original images are input and sequentially passed through an imaging system degradation module, customized according to the hardware parameters of the system, and input into a performance awareness module to obtain the final target acquisition performance. Moreover, to verify the effectiveness of the model, we realized a self-built infrared polarization dataset for sea surface scenes based on infrared radiation theory, and trained and tested the networks. The results obtained when the model was applied to evaluate the performance of infrared polarization imaging systems showed good agreement with subjective perception.