Severval camera pixels are usually corresponding to one projector pixel in three-dimensional (3D) imaging system using binary spatiotemporal encoded illumination. A novel method is proposed for subpixel image matching of camera and projector to increase the measurement precision and data density. The phase distribution of image captured by camera is obtained with sinusoid fitting on temporal direction, according to the binary spatiotemporal encoding strategy and the low pass filter characteristic of two-dimensional transfer function of measurement system. Because of the direct ratio of the phase distribution and projector pixel the corresponding point pairs between the image of projector and camera can be identified through the phase distribution with subpixel accuracy. The experimental results show that the measurement precision of the 3D imaging system is increased about one order of magnitude by using proposed method, and its measurement precision is similar with that of the phase measurement profilometry under the same experimental condition.