Accurate, robust, and fast feature representation and automatic registration are urgent needs for far infrared image in aerial scenes. Since the existing Multiple Line Descriptors (MLD) have the problems of “isolated feature” and “limited scale transformation”, thus a feature description method that combines feature points and line descriptors partition statistics is proposed. This paper refers to the feature descriptor as the Segmented MLD (sMLD). Combining the characteristics that sMLD feature connect with each other to form a mesh topology structure, a coarse-to-fine branch accelerated matching (RF-BA) algorithm is also proposed. The RF-BA coarse matching improves the matching efficiency by making full use of the topology structure and local optimal algorithm. The RF-BA fine matching improves the registration accuracy by using the minimum circumscribed convex quadrilateral principle and GMS verification principle. Experimental results show that compared with other existing registration methods, the method has better performance in terms of registration accuracy and running time.