Through femtosecond laser single-pulse experiments, we investigated the influence of polarization directions (0°, 45°, 90°, and 135°) on the damage threshold of x-cut lithium niobate (LN) crystals. We found that the material damage threshold exhibited considerable polarization dependence, with the damage threshold along the 90° polarization direction being lower than those in other polarization directions. Through first-principles calculations, we confirmed that tunneling ionization caused by the femtosecond laser mainly breaks Nb—O bonds. Along the incident polarization angle of 90°, more number of Nb—O bonds are easily broken, thereby reducing the damage threshold in that polarization direction. This study contributes to an in-depth understanding of the ablation of ultrafast lasers on LN crystal surfaces and has an important reference value for the laser preparation of functional devices on LN crystal surfaces.