In order to improve spots centroid detection accuracy of Hartmann-Shack wavefront sensor (HS-WFS) to achieve high-precision of wavefront test of optical system, a centroid detecting method is proposed. The nonlinear filtering and the windowing spot image in global processing are used, combining with median filtering, cubic spline interpolation and adaptive Otsu thresholding for a single spot in local processing. The regulation of centriod error and calculation time with different interpolation points is discussed. For a spot image with noise, the centroid detection error is only 0.0442 pixel. Compared with the traditional methods of nonlinear filtering, the Otsu thresholding method and the detection windowing, the accuracies are increased by 91.86%, 87.97% and 31.79%. The known wavefront aberration of an optical system is simulation tested. The wavefront testing accuracy is 0.0098 λ in peak-valley (PV) value, and is 0.0027 λ in root-mean-square (RMS) value. The results show that this method enhances the accuracy of centroid detection, and is suitable for high-precision optical systems.