Research Progress in Non-Intrusive Three-Dimensional Reconstruction of Transparent Rigid Bodies

Chifai Pan; Rui Chen; Changping Hu; Jing Xu

doi:10.3788/LOP223415

Laser & Optoelectronics Progress, Volume. 60, Issue 8, 0811004(2023)

Research Progress in Non-Intrusive Three-Dimensional Reconstruction of Transparent Rigid Bodies

Chifai Pan^1、†, Rui Chen^1、†, Changping Hu, and Jing Xu^*

Author Affiliations

Department of Mechanical Engineering, Tsinghua University, Beijing 100083, China

show less

Abstract Get PDF(in Chinese)

Figures & Tables(8)

Fig. 1. General taxonomy of 3D reconstruction of a transparent rigid body

Download full size

Fig. 2. Principle of using silhouette to reconstruct the visual hull (two-dimensional case)

Download full size

Fig. 3. RGB-D camera and the RGB image and depth map from it. (a) RealSense D415 RGB-D camera; (b) RGB image containing opaque and transparent objects from RealSense D415; (c) the depth information obtained by RealSense D415, in which the depth of transparent object is missing

Download full size

Fig. 4. Principle of rendering for the neural radiance field

Download full size

Table 1. Comparison of RGB-D depth completion methods

View table

Table 1. Comparison of RGB-D depth completion methods

Label	Method	Depth completion performance	Inference speed	GPU memory occupation performance
A	ClearGrasp^［28］			>B
B	LIDF^［42］	>A（better than ClearGrasp^［28］）	>C
C	TransparentNet^［32］	>A	>A	>A
D	DepthGrasp^［38］	>A
E	DFNet^［40］	>A	>B	>C
F	A4T^［34］	>A
G	TODE-Trans^［35］	>B，E
H	SwinDRNet^［33］	>B

Table 2. Comparison of reconstruction accuracy between the traditional visual-hull-based optimization method and deep-learning-based optimization method
View table
Table 2. Comparison of reconstruction accuracy between the traditional visual-hull-based optimization method and deep-learning-based optimization method
Object Chamfer distance /mm
Traditional visual hull based optimization Deep learning based optimization
Mouse 0.804 0.535
Pig 0.558 0.487
Dog 0.225 0.186

Table 3. List for datasets of transparent object

View table

Table 3. List for datasets of transparent object

Name of the dataset	Number of frames	Type of data	Number of objects	Type of material
TOD^［50］	40000（real）	RGB image，raw depth，GT（ground truth）depth，pose，and segmentation mask	15	Transparent
ClearGrasp^［28］	50000（synthetic）+286（real）	RGB image，GT depth，surface normal，pose，and semantic segmentation	9（synthetic）+10（real）	Transparent
Omniverse^［42］	60000（synthetic）	RGB image，GT depth，2D/3D bounding box，pose，and segmentation mask	Unknown	Transparent，opaque
TODD^［32］	15000（real）	RGB image，raw depth，GT depth，pose，and segmentation mask	6	Transparent
TransCG^［40］	58000（real）	RGB image，GT depth，surface normal，pose，and segmentation mask	51	Transparent，translucent，and opaque
ClearPose^［49］	350000（real）	RGB image，GT depth，surface normal，pose，and segmentation mask	63	Transparent，translucent，and opaque
DREDS^［33］	130000（synthetic）	RGB image，raw depth，and GT depth	1861	Transparent，opaque
STD^［33］	27000（real）	RGB image，raw depth，GT depth，segmentation mask，normalized object coordinate space map	50	Transparent，opaque

Table 4. Comparison between different reconstruction methods

View table

Table 4. Comparison between different reconstruction methods

Method	Error /mm	Time	Strength	Shortcoming	Application
Visual hull based	0.10	Longer than one hour	High-accuracy reconstruction	Slow reconstruction speed and complicated setup	Measurement
Deep learning based	10	Real time	Fast reconstruction speed，simple setup	Low-accuracy reconstruction，high-cost dataset	Robot manipulation and pose estimation

Tools

Get Citation

Copy Citation Text

Chifai Pan, Rui Chen, Changping Hu, Jing Xu. Research Progress in Non-Intrusive Three-Dimensional Reconstruction of Transparent Rigid Bodies[J]. Laser & Optoelectronics Progress, 2023, 60(8): 0811004

Download Citation

EndNote(RIS)BibTex Plain Text

Set citation alerts for article

Save article for my favorites