Aiming at the problems of poor robustness of rotation change, high feature dimension, and long retrieval time of existing color image retrieval algorithms, this paper proposed an innovative image retrieval method by fusing color features and improved directional gradient features. It proposed an improved directional gradient algorithm based on the principal curvatures (P-FHOG) by combining the geometric curvature information of the image surface into the FHOG descriptor from multiple scales. At the same time, the color information of the image was further fused to obtain the multi-scale image retrieval method based on the color features and the improved directional gradient features (CP-FHOG). The experiment was compared with the advanced image retrieval methods on the Corel-1000 and Coil-100 data sets, and the average accuracy rates of 85.89% and 93.38% were achieved, respectively. The results show that the proposed algorithm is more accurate and robust (in rotation change) than other algorithms.