The position and orientation measurement system based on computer vision is extensively applied on robotics, motion control and precision detection systems. Using the minimum hardware resource, a localization system based on mono-vision and manual planar target was designed and the method of image matching and position resolving was also studied. Firstly, the object detection based on image matching was used to get the coordinate of the planar target in image. The matching was based on SIFT features and projection estimation to detect the target in image, and then the accurate coordinate of the target′s center was calculated by some inherent shape information. A number of image samples were used to validate the accuracy and robustness of the image matching algorithm. Secondly, a new method to solve the PnP problem based on the rectangular distribution was proposed. The method uses the locations of target control points in the image coordinate system and the object coordinate system was used to get the relative position and orientation between the moving object and the camera. The experiment was conducted on the 5D precision displacement stage and results show that the system can achieve the accuracy level of mm in the range of 800 mm, which meets the project requirement.