According to the characteristics of object edge information and noise in infrared image, a bidirectional order filtering algorithm based on statistical sorting is proposed to find the best balance between denoising and keeping image edge information and small target. The FPGA IP module of bidirectional sequential filtering has been widely used in infrared products. It can effectively filter the salt and pepper noise in infrared images, reduce the fixed residual after nonuniformity correction and the electronic noise of signal processing circuit. The application results show that the proposed algorithm has strong filtering ability and less loss of image details, The designed FPGA IP module has less delay and stable timing performance.