The traditional moving object detection algorithms cannot work when both background and objects are in moving state. A method of infrared image moving object detection is proposed on moving background. First a selecting method of image characteristic block based on frame differences is proposed and the improved mutual information measure is combined to match background. Then the matched images are dealt with frame differences and the frame differences image is segmented by the threshold which is gotten based on image characteristic block. The background noise is wiped off by using morphology method. At last, the object detection is completed by image projection. Experiments show that the proposed algorithm can eliminate background motion effectively and detect the moving objects on the moving background accurately.