In actual welding operations, due to the interference of complex environmental factors such as strong optical noise, spark splashing, and smoke, traditional weld seam detection methods relying on a single tracker often find it difficult to maintain accurate tracking of the weld seam, resulting in a significant decrease or even failure in tracking performance. Therefore, a real-time tracking and detection method for infrared images of weld surface defects based on U-net neural network is proposed. This method accurately identifies the weld seam features in the infrared image of the weld seam surface during the initial stage of welding, and precisely locates the feature points. In order to cope with the strong interference environment during the welding process, two parallel kernel correlation filters are designed to track the weld seam feature points, and the output results of these two trackers are fused through a Kalman filter to ensure real-time, stable, and robust tracking of the weld seam even in complex environments. Real time tracking of weld seam feature point information is used as a key input and fed into the U-net neural network. In the U-net architecture, a branch network is introduced to optimize the feature extraction process and improve the quality of the segmentation map, enhancing the ability to capture details of surface defects on the weld seam. Using the bounding box mechanism to analyze the segmentation map output by U-net, automatic determination of the position and size of defect areas is achieved, and infrared image detection of surface defects in welds is completed. The experimental results show that this method performs well in both weld seam tracking and infrared image detection of weld surface defects, with an evaluation function Q value as low as 21.36, indicating high detection accuracy.