Nuclear evaporator has two sleeves, outside sleeve and inside sleeve. To install inside sleeve accurately, the concentricity of the two sleeves and the roundness error of inside sleeve should be evaluated. The system which contributes to install inside sleeve of nuclear evaporator is developed. It is introduced that the parts of the system and the measuring process. The position forms among three close points are divided into three forms: convex, concave and collinear in the data processing. And it is found out how to filter the points that can′t fix maximum inscribed circle. Based on that, the filtering algorithm is presented. This algorithm is efficient and easy to be understood. It can save the data processing time. In practical application, the measuring accuracy of the whole system is 0.7 mm, which can meet the needs of project and have practical value.