An active disturbance rejection controller for an electro-optical tracking servo system is presented to track maneuvering targets fast and accurately and then its work principle is investigated. Based on the structure analysis of a tracking system, the target velocity is taken as the exterior unknown disturbance and the system parameter change is used as the interior unknown disturbance. A extend state observer and a tracking differentiator are used to estimate the disturbance accurately from the input and output information of system response, then the controller can compensate the disturbance further. Experimental results indicate that the system can fast track maneuvering targets at a maximal angular velocity of 40 mrad/s and an maximal angular acceleration of 8 mrad/s2 while the maximal tracking error and the maximal overshoot are less than 0.1 mrad and 10% respectively.These results show that the scheme has a good structure and robust performance, and can satisfy the system requirements without using the sensor to measure the target state.