Aiming at the autonomous decision-making problem of spacecraft approaching closely in the Earth-Moon environment， a decision-making method based on an improved Proximal Policy Optimization （PPO） algorithm was proposed to enable the tracking spacecraft to reach the state required for docking with the target spacecraft within a specified time. First， an LSTM network was introduced into the strategic network structure of the PPO algorithm to handle state inputs and increase the robustness of the algorithm in learning tasks with random parameters. Secondly， a state-based internal reward exploration mechanism was proposed to improve the algorithm's exploration ability by linearly superimposing it with the algorithm's basic reward. In addition， an importance sampling ratio constraint was designed and introduced into the strategy loss function to prevent high variance objective estimation from endangering the optimization of the objective function. Finally， the effectiveness of the proposed method was verified by comparing the learning reward and task execution results with other learning algorithms. The simulation results show that the learning reward value of the improved PPO algorithm is increased by 15%， the fuel consumption of performing close tasks is reduced by 57%， and the mission success rate is increased by 1% when there is unmodelled interference. This method can significantly improve the spacecraft's autonomous decision-making capabilities when performing close missions.