Traditional forward design process of metasurface optical filters is computationally costly and time-consuming; therefore, the inverse design based on deep learning can help accelerate the process. Here, we propose Global- and Local-Spectrum-aware Transformers (GLSaT), a deep learning model that concerns the intrinsic correlations within the spectral sequences, compensating the drawbacks of current networks that only focus on structure-to-spectrum mappings. With both inter- and intra-fragment attention mechanisms implemented, the GLSaT achieves 32.9% higher accuracy than fully connected networks in our reflection tests. It also demonstrates an inherent balance between predictive precision and computational efficiency, outperforming alternative architectures. Furthermore, our extensive experimental validations demonstrate its generalization capability across diverse metasurface functionalities. The GLSaT architecture shows great potential for enhancing the efficiency of data-driven metasurface inverse design in the future.