In order to recover depth information from two-dimensional color image, a visual-dictionary-based depth map generation algorithm is proposed. A data-driven method is used to find depth information of various spatial structures from depth map library, so as to obtain initial visual words which consist of image patches with similar structure. Hard example mining method is used to find hard negative examples of visual word, and visual word classifier is updated to get best classification result. Visual dictionary composed of visual word classifiers and visual words is used to detect target image at multiple scales to get corresponding depth map, to which edge-preserving smoothing filter will be applied. Experimental results show that depth maps generated by the proposed algorithm match depth change of target images, and has a good improvement in both subjective visual effects and objective evaluation indexes.