Rivet flushness is an important indicator of riveting quality parameters; however, efficient and stable methods for actual testing are lacking. We propose a technique for flushness detection based on image-to-point-cloud segmentation algorithm for detecting rivet flushness. First, we propose a separating method of the image noise contour to stably and quickly extract the rivet contour in images. Three neighborhood features at the inflection point of the contour are summarized on the basis of an analysis of the neighborhood features of the rivet contour pixel. According to the neighborhood features, whether the contour point is an inflection point is judged, and the noise contour separation is realized. Second, the rivets features in the image are mapped to the measured three-dimensional point cloud to realize fast and stable segmentation of the rivets after extracting the contours of the rivets in the image. The experiment confirms the excellent stability and accuracy of the rivet flushness detection method proposed in this paper.