The fixed-size patch division method may cause the local structure of an image to collapse and induce inconsistency in semantics between images during visual Transformer-based hyperspectral unmixing. Therefore, this study proposes a data-driven method that adaptively adjusts the position and scale of patch division according to the input hyperspectral image and effectively captures the local features and spatial positions in the image via a multiscale deformation attention mechanism. In addition, a multilevel feature extraction strategy is used to comprehensively mine the rich features contained in the image. A comparison of experimental results obtained for multiple real datasets shows that the proposed method can accurately estimate the abundance map and endmember spectrum. Moreover, it exhibits excellent generalization ability and stability in different scenarios, especially during the spectral feature processing of complex objects.