We present an experimental and theoretical study of self-rotation of optical polarization in a rubidium vapor. The atomic vapor is placed in a magnetic shielding cavity to suppress the Faraday rotation effect. In our experiment, Doppler-free spectroscopy configuration is used, and F=2 -> F'=3 transition of 87Rb D2 line is chosen. We observe self-rotation of optical polarization effect at different pump light ellipticities. A theoretical analysis is then provided based on the experimental conditions. Theoretical simulation and experimental results are in good agreement.