Aiming at the difficulty of a single method to balance the global and local information to accurately extract the edges of non-cooperative targets in the field environment, combined clustering and other algorithms a new method of edge extraction was proposed. Firstly, according to the image pixel value clustering, each two categories were divided by threshold to obtain a binary image and the binary images were subjected to or operation. Then, after the image was inverted, the connected domain with the largest area was retained to obtain the target segmentation image, and the target edge was extracted. Finally, sub-pixel edge calculation based on Zernike moment was processed. The new edge extraction method had strong adaptability, and could quickly and effectively extract the target edge in the actual environment. The non-cooperative targets in the experiment were the three inner holes of the device. The sub-pixel edges were extracted by the above method, and then the center of the circle was fitted, and the center was used for relative pose measurement. The experimental results show that the method is robust and accurate. The maximum position deviation is 0.12 mm, the measurement precision of the attitude angle which is perpendicular to the optical can reach 0.02°, and the measurement precision of the other two attitude angles can reach 0.07° and 0.08°.