The response nonuniformity of uncooled infrared focal plane arrays is very similar to the stripe noise. Scene-based nonuniformity correction algorithm is an effective way to enhance the image quality and compensate the response drift. Based on the study of moment matching theory, a novel temporal moment matching nonuniformity correction algorithm is proposed. In the proposed algorithm, changing column is identified by the adjacent frames after moment matching, the correction coefficients are updated adaptively in temporal domain. The experimental results with the real infrared video sequences have shown that the proposed algorithm can significantly increase the convergence speed and reduce the ghosting artifacts.