The accuracy displacement measurement system based on an optical joint transform correlator has widely potential application for its high speed and parallel computing.But its accuracy is restricted because of the centroid subpixel error. An optimal correlation peak extraction algorithm and a compensation algorithm, which can be applied to the practical displacement measurement with the minor systematic error, are presented. Though simulation, experimental analysis and comparing the subpixel error effects based on weighted center of mass (COM), paraboloid fitting algorithms, Gaussian fitting, the best correlation peak extraction algorithm is COM with a weight degree of 2. The experimental data indicates that the residual error is declined from ±0.1 pixel to ±0.04 pixel through the new compensation algorithm, and the effection of subpixel error of the system is declined accordingly, and the displacement measurement accuracy of an optical joint transform correlator is improved.