For the problem of infrared video in maneuvering target recognition such as high noise and low contrast, an improvement approach based on Median filter algorithm is put forward to image preprocessing. On this basis, a method using the description of adaptive change background pixel number of Gauss distribution is presented to reduce background modeling time, and a new algorithm which combines the background difference method with the frame difference method is used to extract the maneuvering targets. When multiple maneuvering targets are detected, an algorithm based on the coordinates of the center of gravity connected domain mark is applied to locate the target area. Then Haar_like rectangular feature is selected to extract target features. Finally, maneuvering target recognition is achieved through supported vector machine (SVM). Experimental results show that the proposed algorithm can efficiently identify people and vehicles in the infrared video in real time.