The problem of infrared image target recognition based on classifier decision fusion was proposed. The sparse representation-based classification (SRC) and convolutional neural network (CNN) were used as the basic classifiers. For the test sample, it was first classified based on SRC, and the reliability of the decision was judged based on the output decision variables. When it was determined that the recognition result is reliable, the recognition process ended and the target category was output. On the contrary, some candidate categories with higher confidence were selected according to the results of SRC, and CNN was employed to confirm the classification result in the next stage. In addition, the CNN output result and SRC were subjected to linear weighted fusion processing, and the final target category decision was made according to the fusion result. The proposed method integrated the advantages of both SRC and CNN classifiers to comprehensively improve the performance of infrared target recognition. At the same time, this hierarchical decision fusion method avoided the two classification processes for all samples, and could ensure the overall efficiency of the recognition algorithm. The experiment was carried out using five types of infrared images of common vehicle targets in daily life, and the original sample conditions, noise sample conditions and occlusion sample conditions were set respectively. By comparing with some existing methods, the results reflect the effectiveness and reliability of the proposed method.