The flaw in most depth videos is the temporal inconsistency of the depth value of static region, which decreases encoding performance and the quality of rendering. To solve the problem, this paper proposes a method to enhance the temporal consistency of depth video based on the motion intensity of scene. Firstly, we applied block histogram difference (BH) to measure the relative motion intensity of each depth frame, and selected a segment of video as the source for refinement adaptively according to BH value. Secondly, we detected motion region and made a segmentation for each frame of corresponding color video，then refined the depth value which had changed variously in static regions using the accurate temporal consistency information of color video. Finally, we applied the weighted mode temporal filtering on refined depth video to generate well optimized depth video further. Experiment results show that proposed algorithm can save encoding bite rate ranging from 17.48% to 31.75%, while it improves the subjective quality of rendered virtual views.