Ghost imaging has shown promise for tracking the moving target, but the requirement for adequate measurements per motion frame and the image-dependent tracking process limit its applications. Here, we propose a background-removal differential model based on asymptotic gradient patterns to capture the displacement of a translational target directly from single-pixel measurements. By staring at the located imaging region, we evenly distribute optimally ordered Hadamard basis patterns to each frame, thus the image can be recovered from fewer intra-frame measurements with gradually improved quality. This makes ghost tracking and imaging more suitable for practical applications.