A novel speckle reduction technique for digital holography is proposed. Multiple off-axis holograms are recorded using a circularly polarized illumination beam and a rotating linearly polarized reference beam. The speckle noise in the reconstructed images is suppressed by averaging these fields. We demonstrate the effectiveness of this technique experimentally and conduct additional statistical evaluation.