It is a key problem to calculate the rotation angles of the two prisms through the required pointing position of beams when a rotational double prism beam steering system is used to control the direction of optical beams (named inverse problem). This paper explores the entire analytical solutions of the reverse problem for the rotational double prism beam steering system by employing a first-order paraxial approximation method and a non paraxial ray tracing method. First, the centering algorithm is adopted to analyze the approximate solutions of the reverse problem based on first-order paraxial theory. Then, the nonparaxial ray is traced in the system based on Snell's law with the vector form and a two-step algorithm is applied to calculation of the accurate solutions of the inverse problem. The difference of the inverse solutions calculated with the two methods is analyzed and an experiment is designed to compare and validate the solutions. The results indicate that there are two reverse solutions for an required beam pointing position. Relative to the first-order paraxial approximation method, the reverse solutions derived from the nonparaxial ray tracing method are more accurate. For the rotational double prism beam steering system with large beam deviation, the nonparaxial ray tracing method has a potent effect on calculation of accurate reverse solutions.