Vibration is a primary detrimental effect generated by blasting operations, and accurately evaluating its impact remains crucial and challenging. Based on the blasting excavation of a tunnel under the Central Yunnan Water Diversion Project, this study combines numerical simulation and field investigation to assess the damage characteristics of buildings affected by various factors. The results show that blasting vibration causes “X-shaped” cracks at the four corners of windows and doorways, while uneven settlement leads to 45° diagonal cracks. Subsequently, time-frequency analysis was performed on vibration data from buildings at varying distances from the blast source. The findings indicate that forced vibration predominates in building foundations, with minimal free vibration and quickly attenuating after the blasting load ends. As horizontal distance increases, the main frequency and blasting vibration energy exhibit a downward trend based on Fast Fourier Transform (FFT) analysis. However, the main frequency is less sensitive to distance changes than energy. Additionally, the sensitivity of energy to distance varies across different frequency bands. Generally, energy in each frequency band rapidly attenuates close to the blast source, with slower attenuation as distance increases. Furthermore, as the distance from the last source increases, there is a shift in energy from higher to lower frequency bands towards lower frequency bands, and the effect of low-pass and high-filter results in distinct variations in energy attenuation within different frequency bands. Finally, the study highlights a significant disparity between human perception of blasting vibrations and building safety standards. Based on this observation, a comprehensive evaluation method is proposed to combine structural damage assessment with considerations of the human settlement environment.