Rabi model has been a hot topic in recent years. In this paper, quantum phase transitions and multistable states in Rabi model are presented under multiple-photon transition. By means of spin-coherent-state variational method, the equivalent pseudospin Hamiltonian is diagonalized and the energy functional is obtained. Then according to the results of the variational method, the photon number solutions and the critical values are given. Finally, phase diagrams areplotted based on the photon number solutions and the stabilities. The main results are what effects do the numbers of transitions photons have on quantum states, multistability and quantum phase transitions in Rabi model. The results are helpful to induce interesting quantum phase transition by modulating the frequency of the optical cavity.