The relation between the prisms′ orientations and the corresponding pointing positions of an outgoing beam should be figured out when a beam steering mechanism was used to steer the direction of optical beam. Therefore, this paper explored the beam steering mechanism of a rotational double-prism pointing system by applying first-order paraxial approximation method and nonparaxial ray tracing method. Then, it calculated the analytic formulae of the pointing position for the outgoing beam based on the prisms′ rotational angles. The results obtained with the two methods were compared and validated by designed beam steering experiments of rotational double prisms. The results show that the nonparaxial ray tracing method can describe accurately the beam steering mechanism, while the results obtained with the conventional first-order paraxial approximation method has a difference from the experiment value. The larger the beam′s angular deviation is, the more obvious the difference between the solutions with first-order paraxial approximation method and the corresponding experiment values is. When the difference of the prisms′ rotational angles is 90°, the difference between the solutions of azimuth with first-order paraxial approximation method and the corresponding experiment values becomes a maximum one. It suggests that the nonparaxial ray tracing method is suitable for discussing the beam steering mechanism for the rotating double-prism beam steering system with a large angular deviation.