A novel local invariant feature based image registration method for color image is proposed. In the stage of feature point extraction, a new method named colored difference of features from accelerated segment test (CDoFAST) is proposed. The color invariant value of the image is calculated, and FAST extreme points in scale space are searched. The difference of Gaussian (DoG) value around the extreme points are interpolated and fitted to determine the location and scale of the feature points. In the stage of feature vector extraction, a new colored binary local invariant descriptor (CBLID) is proposed. Its sample pattern is similar to the human visual overlap. By generating binary chain code using the statistics of orientation maps, the descriptor is invariant to rotation, scaling, illumination changes and is robust to noise. The feature vectors are matched by calculating their hamming distance and eliminating wrong matches by random sample consensus (RANSAC). Then the transform matrix between the reference image and the registered image is calculated. The experimental results indicate that the proposed method outperforms other classical methods such as scale invariant feature transform (SIFT), speed up robust feature (SURF) and DAISY in registration accuracy and cost time. The cost time of the proposed method in processing the experimental images are only 10% and 12% of that cost in SIFT.