Hyperspectral unmixing is the process of extracting endmembers and abundance features through image decomposition. However, intra-spectral variability caused by factors such as illumination and atmosphere, or in-ter-spectral variability caused by non-linear factors such as environmental changes and equipment, can lead to a decrease in feature extraction accuracy. To comprehensively consider the issue of spectral changes during the unmixing process, an enhanced spectral unmixing optimization model is proposed in this paper by introducing a low-rank orthogonal prior for spectral variability. Firstly, a variability data fitting term is introduced on top of the linear unmixing model to account for both intra-class and inter-class spectral variations. And a scaling factor is used to address intra-class variability in the spectrum, while a spectral variability perturbation matrix is added to address inter-class variability. Secondly, the model utilises orthogonal prior constraints to achieve the low spatial coherence between the original spectral dictionary and the variability term, and suppresses tiny tiny components and noise by employing kernel norm logarithmic relaxation to strengthen the low-rank property of the abundance matrix. Finally, the alternating optimization method and vector-matrix operator are used to reduce the complexity of the solution algorithm. The results of simulation tests on both simulated and real datasets show that the proposed algorithm achieves better performance than the comparison algorithm, which verifies the effectiveness of the optimization model.