Augmented reality technology can superimpose virtual information onto the real environment, which has a broad application prospect in the field of aircraft assembly. At present, aircraft piping and cables still mainly rely on manual assembly, usually using 2D drawings or 3D models for guidance. However, 2D drawings are usually difficult to accurately display complex assembly details, which can easily lead to misunderstandings or omissions. Furthermore, computers are usually placed in designated areas outside the cabin, so that workers need to interrupt the assembly work to view the 3D model, resulting in low assembly efficiency. With the help of augmented reality technology, the 3D model, process information, and other virtual content can be directly projected into the actual assembly scene, which provides workers with 3D visualization guidance and reduces the difficulty of their understanding. Subsequently, the augmented reality technology can be applied to the assembly quality inspection process, thus significantly improving production efficiency. Virtual-real registration is a key technology in augmented reality application, which determines the accuracy of virtual-real object alignment, and commonly used virtual-real registration methods such as sign-based, model recognition-based, and human-computer interactive. However, these methods make it difficult to realize the accurate registration of large-size and structurally complex aircraft parts, which results in the great limitation of augmented reality-based assembly guidance technology in the application of the industrial field. Therefore, we propose a multi-point augmented reality registration method for aircraft pipeline cable assembly, which combines target probe design and calibration, SVD-based positional transform solution and World Locking Tools (WLT) augmented reality space precision locking to effectively improve the accuracy of the augmented reality alignment of large-size parts.

First, we design a handheld target probe and calibrate it to determine the coordinates of the tip point of the probe under the target coordinate system, achieving a more accurate measurement of the point on the surface of the part. Second, we conduct the multi-point virtual-real registration based on SVD. The target mark on the probe uses a QR code, and the size is designed to be 10 cm×10 cm. Different probe tips are designed according to the typical features of aircraft parts, including universal needle probes, through-hole probes, and chamfered probes. The probe calibration system uses an industrial camera and rotary calibration is conducted around the tip of the probe. The probe calibration process target needs to always be in the field of view of the camera. The probe calibration principle is the same for different probes, namely that in the probe rotating process, the probe tip coordinates in the camera coordinate system are always unchanged. Unity is adopted to develop the multi-point registration program, specifically relating to QR code recognition, registration point distribution, singular value decomposition for attitude transformation, WLT virtual-real space alignment method development, human-computer interaction, virtual scene content layout, and other content development. After deploying the developed program into HoloLens2 glasses and running the developed App, firstly, we select the suitable registration points on the virtual model. Then we use the target probe to select the corresponding points on the real parts and utilize the singular value matrix decomposition method to solve the positional transformation between these two groups of points in the virtual-real mapping space of the augmented reality device. Finally, the 3D model can be aligned to the real parts according to this transformation.

We experimentally verify the effectiveness of the target probe calibration (Tables 1 and 2). Under the target coordinate system, the standard deviation of the tip coordinates in the three axes of x, y, and z are all less than 1 mm, and the calibration results have a high degree of stability. From HoloLens2 single-point repeatability experiments (Table 3), we can see that the standard deviation of the three axial directions is not more than 1.8 mm and the error in the process of large-size parts of the virtual-real registration process is tiny, verifying that the designed target probe meets the needs of the use of multi-point registration method. To quantify the virtual registration accuracy of large-size parts, we apply the target probe to test the virtual-real alignment accuracy of the wing within the range of 3 m×1.4 m×0.5 m. The experimental results show that the absolute accuracy of the virtual-real registration can be better than 3 mm, which meets the needs of actual aircraft pipelines and cables for augmented-reality assembly guidance applications. We also analyze the impact of the number of registration points and layout on the accuracy of virtual-real registration based on experiments. We carry out experiments in turn with 3, 4, 5, 6, 7, 8, and 9 pairs of registration points (Fig. 18, Fig. 19), and the experimental results show that the registration error and the RMSE are larger when 3 pairs of registration points are used. With the increase in the number of registration points, the registration error gradually decreases and tends to be stable, and the RMSE declines insignificantly and remains stable. Therefore, it is more appropriate to choose at least 4 pairs of registration points for the validation object of this paper. The distribution of registration points is also analyzed experimentally (Fig. 21), and the results show that when the coverage of registration points is less than 30%, the registration error and RMSE are larger, and there is unstable registration. With the increase in the coverage of registration points, the registration error and RMSE show a gradual trend of decreasing. Thus the distribution of the registration points tries to cover the whole model as much as possible. In conclusion, our method can realize the accurate virtual registration of large-size 3D models, and the number of registration points can be adjusted according to the size of the parts. At the same time, we need to consider the problem of low registration efficiency caused by too many pairs of registration points, and the distribution of registration points covers the whole model and is not in the same plane as much as possible.

Our multi-point registration method can effectively improve the accuracy of virtual-real alignment of large-size parts, and the method also has the advantages of lower algorithmic computation and no special requirements for the structure of the object to be virtual-real aligned. It should be noted that this method partially relies on the more stable spatial localization technology of HoloLens2, and subsequent research will continue to improve the spatial localization accuracy and stability of augmented reality devices to further improve the stability of the virtual-real registration. In addition, the accuracy of the probe calibration is very critical to the subsequent registration and accuracy verification, and higher precision probes can be designed or their calibration algorithms can be improved to further improve the accuracy of the virtual-real registration.