Second-order Local Derivative Pattern (LDP) is a general framework to encode directional pattern features based on local first-order derivative variations. Different from Local Binary Pattern (LBP), the second-order derivative pattern extracts local information by encoding various distinctive spatial relationships contained in a given local region and thus it can capture more detailed information than the first-order local pattern used in LBP. A new algorithm based on Fusion of Second-order Local Derivative Pattern in Gabor characteristic (FG2LDP) for face recognition is proposed. According to the good spatial position and orientation of Gabor filter, a Gabor filter with four frequencies and six orientations is firstly applied to filter face images. Secondly, the weight of each frequency and orientation is adaptively estimated for subsequent fusion. Thirdly, the second-order local derivative information of filtered images is extracted and low dimensional features in every direction are extracted by Principal Component Analysis (PCA). Finally, all the likelihoods in every frequency and orientation are fused together for the final recognition result. Experiments show that our method can effectively extract local features. It can consistently outperform other recognition methods based on PCA, Linear Discriminant Analysis (LDA), LBP and Fusion of Second-order Local Derivative Pattern (F2LDP) in gray images.