Calibration methods of traditional star sensors ignore the additional errors from the interaction between optical parameters and distortion coefficients. This paper proposes an autonomous calibration algorithm based on nonlinear optimization to overcome the problems mentioned above. Firstly, the algorithm ignores the distortion to construct a target function, and the Levenberg-Marquardt nonlinear optimization algorithm is used to optimize the optical parameters of the star sensor. Then, the optimized optical parameter estimation iss used as the ideal value, and the lens distortion coefficient of the camera is estimated by the linear least square method. Finally, the parameters obtained by the first two steps are used as initial values to construct the target function, and the optical parameters and distortion coefficients are optimized by using Levenberg-Marquardt algorithm. Simulation and comparison experiments are performed in combination with least square method and Samman method, and results show that the maximum residual obtained by the algorithm is 0.015 pixels under the same test condition, and the accuracy is higher two orders of magnitude than that of the other two calibration methods. Moreover, the field experiments show that the proposed method effectively improves the performance of star sensors.