To balance the trade-off between tracking accuracy and model complexity, an efficient single-object tracking method is proposed based on a siamese network. The method employs a lightweight MobileNet-V3 as the backbone network, significantly reducing the computational load and number of parameters for feature extraction. Additionally, a hybrid feature fusion module is designed, comprising a rapid feature refinement unit and a dual-branch feature aggregation unit. The rapid feature refinement unit effectively decreases the number of feature vectors by aggregating queries and optimizing keys, thereby quickly extracting key information about the target object. The dual-branch feature aggregation unit further enhances tracking performance through a multi-head attention mechanism that fuses features from different branches. Comparative experiments with other tracking algorithms are conducted on the LaSOT, OTB100, and UAV123 datasets. Experimental results demonstrate that the proposed method maintains satisfactory tracking performance while exhibiting lower model complexity. Furthermore, it sustains robust tracking capabilities in various complex scenarios, including fast motion and rotation.