Infrared images and visible light images record different aspects of the nature of a ground object, such that the fusion of two such images of the same object can compensate for a lack of information from a single data source. However, due to the distinct imaging principles involved, the difference between the same-scene images produced by a gray image sensor and a visible light sensor is large, resulting in mismatched images that are difficult to fuse. In this paper, a matching method based on the analysis of the common features of infrared and visible light images using SIFT and ORB feature detection is proposed. The SIFT operator and the ORB operator are used to simultaneously perform feature point detection. First, the same name is obtained, using RANSAC, for SIFT matching. The points are filtered, and the nearest neighbor neighboring nearest neighbor algorithm is used to obtain the ORB matching points. Then the SIFT matching points are used to geometrically constrain the distance and angle of the ORB matching points to further reduce the mismatch. Ultimately, the feature points are evenly distributed and the reliability is higher, solving the poor-matching-effect problem. The performance of the proposed method was compared with that of SIFT using four sets of infrared and visible images, with the proposed method achieving a number of correct matching feature points approximately 3.7 times, 3.2 times, 3.6 times, and 3 times higher than those achieved with SIFT. This significant performance improvement indicates the effectiveness of the proposed method.