To solve such problems in infrared image pedestrian detection as lack of image details, low detection accuracy and the difficulty in extracting the feature information, an infrared image pedestrian detection algorithm based on the improved SSD algorithm was proposed.Depthwise Separable Convolution (DSC) method was used to reduce the number of parameters and calculation costs of the network in feature extraction, and improve the real-time performance.The SENet module was embedded in the network, and the weight of each feature channel was reassigned, so that the network could better target the pedestrians. Since the space ratio of the pedestrian target is relatively stable, the Default Boxes value was set by using cluster analysis algorithm, so as to improve pedestrian detection effects.Experimental results showed that, the proposed algorithm performed better than the VSSD algorithm.The precision ratio and the recall ratio reached 91.7% and 84.8% respectively, and the real-time performance of the algorithm was also greatly improved.