To mitigate the issue of friction disturbance in the photoelectric tracking system and to improve the stability and precision of the line of sight, a compensation technique known as LuGre-TSMO was employed. The LuGre model was combined with a terminal sliding mode observer (TSMO) in this approach. Specifically, a terminal sliding mode observer was devised based on a novel reaching rate, utilizing the LuGre model as a basis. This design enables the system to achieve secondary compensation for friction. In comparison to the scenario without compensation, the root mean square error of speed tracking was reduced by an average of 65.38%. Consequently, the accuracy of visual axis stability on the experimental platform was effectively enhanced. The simulation and experimental outcomes demonstrate that the LuGre-TSMO method exhibits the most effective friction compensation effect when compared to the LuGre model method and the terminal sliding mode observer method. Furthermore, it is demonstrated to be both feasible and effective in the context of the photoelectric tracking system.