The main deficiency of traditional interest points-based image retrieval methods is that the accuracy is influenced by the dissimilar interest points in the non-interest regions. A novel algorithm for image retrieval based on interest points was presented. By comparing the Euclidean distance of each interest point’s local Zernike moments based on its local gray information,the best matching pairs of interest points were computed and the dissimilar points were eliminated. Then,the spatial cohesion of interest points was used to discriminate the similarity between two images. With robustness to rotation and translation,the algorithm avoided the shortcoming of traditional methods. Experimental results show that this algorithm is simple and improves the image retrieval accuracy efficiently.