An adaptive depth image hole inpainting algorithm is proposed herein to address the problem of the depth image being affected by the depth value of external objects during the process of edge inpainting. First, the edges of the depth image and RGB image were extracted, then the false edge caused by holes was removed and the edges were fused. The size of the filter window was then determined using the fused edge information and effective pixel ratio. The weight parameters of spatial distance were set on the basis of the relationship between filter window and spatial distance. The maximum structural similarity between the depth image and RGB image was used as the color weight parameter. Finally, a median filter was used to remove edge hole pixels and discrete noise. The proposed algorithm is tested using the Middle bury dataset and NYU Depth Dataset V2 and compared with other algorithms. The results show that the proposed algorithm can repair holes in the edge area more effectively, maintain clearer object edge contour information, and has high robustness.