When joint bilateral filtering is used to repair depth images, hole-filling effect is poor because the filtering neighborhood range and weight parameter cannot be estimated accurately. To address this problem, we propose an adaptive hole-filling and optimization algorithm for depth images. The proposed algorithm reduces the input parameters and restores each missing depth value. First, the filtering neighborhood range of each hole pixel is determined based on the effective pixel proportion. Then, the parameter value of spatial distance weight is calculated based on the neighborhood size. Finally, the structural similarity is introduced as a parameter calculation index of the color similarity weight. The performance of the proposed algorithm is tested on the Middlebury stereo-matching dataset and the registered Kinect RGB-D dataset, and qualitative comparison and quantitative analysis are performed to compare the performance of the proposed algorithm with those of other methods. The experimental results show that the developed algorithm can effectively fill in missing depth values, reduce the image noise, and improve the quality of depth images meticulously and accurately.