Aiming at the characteristics of complex working mode and high coupling with flight state of rocket based combined cycle (RBCC) power system, a trajectory optimization model for RBCC powered hypersonic vehicle was established. At the same time, the trajectory optimization design framework and solution strategy for RBCC powered aircraft were established based on convex optimization theory. On this basis, the maximum mechanical energy at the end of the rising section was simulated. The simulation results show that the relevant models and trajectory optimization methods are feasible, and the optimization results accord with the working characteristics of RBCC power system. The trajectory optimization method proposed in this paper can effectively deal with the trajectory optimization of RBCC assisted aircraft in the ascending phase under complex working modes, and provides some new ideas for this kind of trajectory design and optimization in the future.