A new, general, fast and user-friendly method to solve the problem of colorizing grayscale images is proposed. The neighborhood-first searching scheme is designed firstly to find the candidate matching pixels of a target grayscale pixel. Then, the final matching pixels are determined by calculating matching error between the candidates and the target pixels based on the texture similarity. For the pixels that don’t meet the neighborhood matching criterion, the whole image searching and matching can be sped up by using edge-preservation smooth filter to smooth the color reference image and setting sampling interval. To address the problem of image distortion occurred after global matching, our method conducts one or more swatch matching procedures to transfer the desired color moods from groups of swatch regions in the color reference image to the distortion regions of the colorized image. The subjective evaluation results show that in four test cases, the proposed algorithm has similar performance to the classical Welsh algorithm on the color transfer quality, but in the aspect of computing efficiency, the proposed algorithm has a great advantage over Welsh algorithm. By means of a test case of colorizing a grayscale image with complex color composition, the improvement of the swatch matching in color transfer and gamut reproduction is verified.