As a cutting tool of rotary drilling rig, the pick teeth need to be analyzed for its cutting performance, and the structural parameters need to be optimized. Based on the discrete element method and response surface experimental design principle, the relationship between the uniaxial compressive strength of rock and its material constitutive model parameters is studied to determine the calibration results of discrete element parameters for four typical rock layers. A multi-objective evaluation model of tooth cutting performance is explored to reveal the cutting performance of the tooth and optimize the inclination angle, skew angle, and tooth spacing of the tooth arrangement. The research results show that the radial arrangement of teeth performs better in soft rock formations, while the circumferential arrangement of teeth performs better in hard rock formations. The cutting performance is best when the radial tooth arrangement has an inclination angle of 45° and a tooth spacing of 56 mm, while the circumferential tooth arrangement has an inclination angle of 30° and a skew angle of 18°. This study can provide reference for the design of tooth arrangement for rotary drilling rigs in different rock formations.