To realize the autonomous optical navigation of planets in deep space exploration, a planetary center measurement method with sub-pixel accuracy was presented. Firstly, the coordinate transformation between a navigation camera and a target planet was established. According to optical imaging theory, the edge characteristics of a planetary image were analyzed. Using the only one semi-circle edge of the planetary image, planetary center was measured through circle fitting. Then, according to the edge distribution characteristics of the planetary image, the real semi-circle edge of planet was extracted by modified Canny algorithm and the longest distance method. Finally, the theory basis of traditional sub-pixel edge detection algorithm was analyzed based on Gauss fitting, improved sub-pixel edge detection algorithm based on Gauss fitting was presented according to imaging theory and the sub-pixel center of planet was obtained through circular curve fitting. In simulation experiment and moon testing experiment, the accuracies of modified algorithm are 0.02 and 0.68 pixels, which are higher about 0.03 and 0.21 pixels than that of traditional algorithm. It can satisfy the requirement of deep space exploration for reliability, positioning accuracy and noise immunity.