The measurement precision of radial velocity needs improving for the detection of terrestrial planets. However, the spectral line shift during the change of coupling resulting from the insufficient scrambling of circular fibers limits the further improvement of the precision. The polygonal core fiber can provide better scrambling properties. Based the built experimental apparatus for measuring the near and far fields of the fiber output, the scrambling performance of the circular, rectangular, square and octagonal fibers under different coupling conditions is characterized via evaluating the near and far field patterns. The intensity variation in the far field and the barycenter shifting in the near field are investigated, and the scrambling gain is obtained for each fiber. The experimental results show that the polygonal core fiber has better scrambling performance. Especially, the octagon fiber has the minimum intensity variation in the far field, the minimum barycenter displacement in the near field, and the best scrambling performance. It is shown that replacing a circular fiber with an octagon one will effectively decrease the spectra line shift from coupling error and improve the measurement precision of the radial velocity.