Optics and Precision Engineering, Volume. 31, Issue 16, 2444(2023)
Review of multi-view stereo reconstruction methods based on deep learning
Figures & Tables(7)
Fig. 2. Classification of MVS Network Improvement Methods Based on Supervised Learning
Fig. 3. Visualized performance comparisons of MVS methods on Tanks and Temples benchmark［106］ （higher is better）
Table 1. Main characteristics and problems of MVS methods with different cost volume regularization strategies
View tableView in ArticleTable 1. Main characteristics and problems of MVS methods with different cost volume regularization strategies
Type Characteristic Problem Method End-to-end CNN Cost volume regularization using 3D CNN Usually slow in training and reasoning， with large memory consumption ［ 18 ］，［70 ］，［57 ］，［55 ］，［71 ］RNN Combining the accuracy of 3D CNN and the efficiency of RNN， greatly reducing memory consumption Reduced memory consumption but increased runtime ［ 16 ］，［47 ］，［72 ］，［73 ］，［41 ］，［74 ］Coarse-to-fine Refine by other methods after obtaining the initial depth map Need some prior knowledge ［ 11 ］，［17 ］，［66 ］，［49 ］，［22 ］，［48 ］，［34 ］，［50 ］，［30 ］，［75 ］，［45 ］，［53 ］，［64 ］，［63 ］，［43 ］，［76 ］Multi-stage Coarse-to-fine Build the cost volume over the entire depth range with coarse resolution， and calculate the reduced sample range based on the coarse depth map Prediction accuracy is highly dependent on the initial depth map. The cost volume characteristics of different stages are not fully considered ［ 60 ］，［61 ］，［27 ］，［20 ］，［62 ］，［19 ］，［24 ］，［54 ］，［51 ］，［31 ］，［67 ］，［42 ］，［68 ］，［77 ］，［78 ］，［23 ］，［79 ］，［39 ］，［28 ］，［32 ］，［65 ］
Table 2. Summary of loss functions of MVS methods based on unsupervised learning
View tableView in ArticleTable 2. Summary of loss functions of MVS methods based on unsupervised learning
Method Loss function Type Network Training methods LSSIM Lsmooth LPC L1 Lfeature View Image gradient LDA Other Unsupervised UnsupMVS［ 89 ］End-to-End √ √ √ MVS2［ 90 ］End-to-End √ √ √ √ √ √ M3VSNet［ 93 ］End-to-End √ √ √ √ Normal depth PatchMVS［ 98 ］End-to-End √ √ √ √ Lgeometric RC-MVS［ 99 ］End-to-End √ √ √ √ Lrender MS-MVS［ 97 ］End-to-End √ √ √ √ Self-supervised JDACS［ 94 ］End-to-End √ √ √ √ LSC Meta MVS［ 91 ］Multi-stage √ √ √ √ SelfsupCVP［ 95 ］Multi-stage √ √ √ Lperception U-MVSNet［ 96 ］Multi-stage √ Optical flow KD-MVS［ 105 ］Multi-stage √ √ √
Table 3. Dataset of MVS methods
View tableView in ArticleTable 3. Dataset of MVS methods
Dataset Scene Resolution Scale Number of scenes Online benchmark Website DTU［ 9 ］indoor 1 600×1 200 pixels 27 097 124 http：//roboimagedata.compute.dtu.dk/ Tanks and Temples［ 106 ］Intermediate outdoor 8-megapixel ≈5600 8 √ https：//www.tanksandtemples.org/ Advanced 6 ETH3D［ 107 ］high resolution indoor and outdoor 24-megapixel 454 13（train） √ https：//www.eth3d.net/ 443 12（test） low resolution 0.4-megapixel 4 796 5（train） 5 212 5（test） BlendedMVS［ 108 ］outdoor 1 536×2 048 pixels 17 818 106（train） https：//github.com/YoYo000/BlendedMVS 7（test） GigaMVS［ 109 ］outdoor gigapixel 3 599 13 http：//www.gigamvs.com/
Table 4. Quantitative results of different MVS methods on DTU datasets［9］ （lower is better）
View tableView in ArticleTable 4. Quantitative results of different MVS methods on DTU datasets［9］ （lower is better）
Type Method Acc.
/mm
Comp.
/mm
Overall
/mm
Method Acc.
/mm
Comp.
/mm
Overall
/mm
Supervised MVSNet［ 11 ］0.396 0.527 0.462 AACVP-MVSNet［ 31 ］0.357 0.326 0.341 R-MVSNet［ 16 ］0.383 0.452 0.417 NR2-Net［ 73 ］0.370 0.332 0.351 P-MVSNet［ 66 ］0.406 0.434 0.420 AA-RMVSNet［ 41 ］0.376 0.339 0.357 Point-MVSNet［ 17 ］0.342 0.411 0.376 EPP-MVSNet［ 67 ］0.413 0.296 0.355 MVSCRF［ 18 ］0.371 0.426 0.398 PatchmatchNet［ 62 ］0.427 0.277 0.352 CVP-MVSNet［ 20 ］0.296 0.406 0.351 DRI-MVSNet［ 28 ］0.432 0.327 0.379 CasMVSNet［ 19 ］0.325 0.385 0.355 IterMVS［ 74 ］0.373 0.354 0.363 UCSNet［ 27 ］0.338 0.349 0.344 SPGNet［ 71 ］0.320 0.382 0.351 BP-MVSNet［ 70 ］0.333 0.320 0.327 D-CasMVSNet［ 42 ］0.348 0.350 0.349 Fast-MVSNet［ 48 ］0.336 0.403 0.370 ASPPMVSNet［ 39 ］0.334 0.360 0.347 PVSNet［ 61 ］0.337 0.315 0.326 Effi-MVS［ 78 ］0.314 0.334 0.324 Vis-MVSNet［ 22 ］0.369 0.361 0.365 GBi-Net［ 75 ］0.315 0.262 0.289 VA-Point-MVSNet［ 49 ］0.359 0.358 0.359 TransMVSNet［ 45 ］0.321 0.289 0.305 PVA-MVSNet［ 24 ］0.379 0.336 0.357 RayMVSNet［ 23 ］0.341 0.319 0.330 Att-MVS［ 83 ］0.383 0.329 0.356 MVSTER［ 53 ］0.350 0.276 0.313 D2HC-RMVSNet［ 47 ］0.395 0.378 0.386 UniMVSNet［ 68 ］0.352 0.278 0.315 MVSNet++［ 57 ］0.407 0.345 0.376 NP-CVP-MVSNet［ 77 ］0.356 0.275 0.315 REDNet［ 72 ］0.456 0.326 0.391 ACINR-MVSNet［ 63 ］0.306 0.364 0.335 CIDER［ 60 ］0.417 0.437 0.427 ADR-MVSNet［ 79 ］0.354 0.317 0.335 LANet［ 34 ］0.32 0.349 0.335 MSCVP-MVSNet［ 65 ］0.379 0.278 0.328 DDR-Net［ 51 ］0.339 0.320 0.329 FPSA-MVSNet［ 32 ］0.363 0.283 0.323 PA-MVSNet［ 30 ］0.313 0.437 0.375 ADIM-MVSNet［ 43 ］0.344 0.298 0.321 DDL-MVS［ 88 ］0.405 0.267 0.336 HSF-MVSNet［ 35 ］0.378 0.353 0.365 BH-RMVSNet［ 84 ］0.368 0.303 0.335 MVSFormer［ 37 ］0.327 0.251 0.289 CER-MVS［ 110 ］0.359 0.305 0.332 CDSFNet［ 46 ］0.352 0.280 0.316 HighRes-MVSNet［ 50 ］0.354 0.393 0.373 MFNet［ 64 ］0.339 0.304 0.321 MVSTR［ 54 ］0.356 0.295 0.326 WT-MVSNet［ 55 ］0.309 0.281 0.295 Unsupervised UNSUP-MVS［ 89 ］0.881 1.073 0.977 SurRF［ 103 ］0.388 0.390 0.389 Meta MVS［ 91 ］0.594 0.779 0.687 CLD-MVS［ 111 ］0.335 0.430 0.383 MVS2［ 90 ］0.760 0.515 0.637 Self-sup-CVP-MVSNet［ 95 ］0.308 0.418 0.363 M3VSNet［ 93 ］0.636 0.531 0.583 JDACS-MS［ 94 ］0.398 0.318 0.358 JDACS［ 94 ］0.571 0.515 0.543 U-MVSNet［ 96 ］0.354 0.354 0.354 PatchMVSNet［ 98 ］0.538 0.365 0.451 RC-MVSNet［ 99 ］0.396 0.295 0.345 MS-MVS［ 97 ］0.383 0.415 0.399 KD-MVS［ 105 ］0.359 0.295 0.327
Tools
Get Citation
Copy Citation Text
Huabiao YAN, Fangqi XU, Lü'er HUANG, Cibo LIU, Chuxin LIN. Review of multi-view stereo reconstruction methods based on deep learning[J]. Optics and Precision Engineering, 2023, 31(16): 2444
Paper Information
Category: Information Sciences
Received: Nov. 14, 2022
Accepted: --
Published Online: Sep. 5, 2023
The Author Email: HUANG Lü'er (9320080310@jxust.edu.cn)
© Copyright 2018-2021 | Chinese Laser Press.
All Rights Reserved 沪ICP备15018463号-20