The performance of object tracking algorithm is usually related to the quality of initial bounding box. In UAV (unmannd aerial vehicle) ground reconnaissance tasks, due to limited response time, it is often difficult for operators to draw accurate initial bounding box, resulting in poor object tracking results. Current bounding box initialization method has some limitations and cannot meet the needs of UAV ground reconnaissance tasks. To meet the demands of actual system,a semi-automatic initialization and optimization strategy was proposed in combination with human subjective choice and visual cognition, which could give example of adaptive optimization algorithm based on visual saliency and salient region segmentation. The strategy was divided into 3 stages: coarse election, adaptive optimization and fine selection. The effectiveness of tracking box optimization algorithm was verified on 2 benchmark datasets. On VisDrone2018-SOT-test-dev dataset, in comparison with before optimization, the average success rate is increased by 0.138, and the highest is increased by 0.262. The average accuracy is increased by 0.135, and the highest is increased by 0.165. On UAVDT (unmanned aerial vehicle detection and tracking) dataset, in comparison with before optimization, the average success rate is increased by 0.093, and the highest is increased by 0.147. The average accuracy is increased by 0.082, and the highest is increased by 0.177. When processing 200×200 pixels image slices, theoretical parallel speed can reach 10 frame/s, which basically meets the real-time requirements. The proposed strategy can be combined with any tracking algorithm and has portability in embedded devices. The main contribution is the discussion of tracking initialization problem and a strategy to improve the accuracy of initial tracking box, rather than algorithmic innovation.