According to the on-orbit working state and imaging characteristics of cameras for space exploration, the lunar-edge method is used to detect its on-orbit modulation transfer function (MTF), combined with the commonly used test method of MTF. This method uses the high-contrast linear edge image formed by the lunar edge and the deep space background as the test target. The simulation is used to analyze the calculation precision and running speed of various processing methods in each link, forming an optimized process of on-orbit MIT detection based on the lunar-edge method. In the link of edge spread function (ESF) processing, a modified Savitsky-Golay filter combined with double Gaussian fitting method is used to obtain an accurate ESF while effectively suppressing the noise. The simulation and experimental results show that the proposed method has an accuracy of 0.02 for images with a signal-to-noise ratio higher than 30 dB, which can achieve accurate calibration of camera focal plane position and MTF.