The presence of the speckle noise greatly reduces the signal-to-noise ratio and spatial resolution of the digital holographic reconstructed image. In the lensless Fourier transform digital holography, the object light field can be reconstructed by only one step of Fourier transform on the interferogram recorded by the charge coupled device (CCD). A new speckle reduction method is proposed based on the principle that each pixel in the hologram contributes differently to the distribution of the speckle noise intensity in the reconstructed image in this paper. With this method, we divide the interferogram outputted by the CCD into several regions and exchange the positions of the regions. Many different new reconstruction interferograms are obtained, and then all new interferograms are reconstructed. Then, the reconstructed image with lower speckle noise intensity can be obtained by superimposing and averaging all of the reconstructed images, and the equivalent number of looks improves significantly. Compared with the spatial domain mask method, the proposed method reduces the processing time by 80%, and increases the edge preservation index of the reconstructed image by one times.