Line of sight (LOS) calibration is the key precondition of accurate location and tracking of targets in infrared surveillance systems and the LOS calibration of scanning sensor is one of the difficulties. Based on the analysis of the imaging model and characteristics of scanning sensor, a theoretical calibration model based on the estimation of mismatching angles is proposed. By establishing the process model of the mismatching angles and the observation model of sensor LOS, and using an unscented Kalman filter (UKF), the real time estimation of mismatching angles and calibration of sensor LOS are finally achieved. Simulation results show that the proposed algorithm has a high precision and smooth performance for sensor LOS calibration, and can effectively improve the timeliness and precision of target tracking process in infrared surveillance system.