The extraction of stable local features directly affects the performance of infrared face recognition algorithms. Recent studies on the application of scale invariant feature transform (SIFT) to infrared face recognition show that star-styled window filter (SWF) can filter out errors incorrectly introduced by SIFT. The current letter proposes an improved filter pattern called Y-styled window filter (YWF) to further eliminate the wrong matches. Compared with SWF, YWF patterns are sparser and do not maintain rotation invariance; thus, they are more suitable to infrared face recognition. Our experimental results demonstrate that a YWF-based averaging window outperforms an SWF-based one in reducing wrong matches, therefore improving the reliability of infrared face recognition systems.