Aiming at the problem of a large number of invalid points in the depth image obtained by the time-of-flight camera, a depth image restoration method based on the color image contour is proposed. This method first determines the variance of the neighborhood through the value of the pixel in the neighborhood of the invalid point and the Gaussian function, and calculates the filling order of all invalid points based on the number and variance of the valid point in the neighborhood of the invalid point, and completes the priority evaluation of the filling process. The contour of the target object in the color image is projected into the depth image, combined with the projection contour and the Euclidean distance function to realize the filling of invalid points. The experimental results show that compared with other commonly used denoising algorithms, the algorithm in this paper can effectively fill invalid points on the premise of protecting the edge of the target object. The signal-to-noise ratio index is improved by 2-14 dB for different target objects.