A tracking technique using two sequentially-connected Kalman filter for tracking laser speckle displacement is presented. One Kalman filter tracks temporal speckle displacement, while another Kalman filter tracks spatial speckle displacement. The temporal Kalman filter provides a prior for the spatial Kalman filter, and the spatial Kalman filter provides measurements for the temporal Kalman filter. The contribution of a prior to estimations of the spatial Kalman filter is analyzed. An optical analysis system was set up to verify the double-Kalman-filter tracker's ability of tracking laser speckle's constant displacement.