Laser & Optoelectronics Progress, Volume. 60, Issue 8, 0811011(2023)

Application of Deep Learning Technology to Photometric Stereo Three-dimensional Reconstruction

Guohui Wang* and Yanting Lu
Author Affiliations
  • School of Optoelectronic Engineering, Xi'an Technological University, Xi'an 710021, Shaanxi, China
  • show less
    References(112)

    [1] Wang G H, Zhang X. Fast shape-from-shading algorithm for 3D reconstruction of hybrid surfaces under perspective projection[J]. Acta Optica Sinica, 41, 1215003(2021).

    [2] Wang G H, Song Y G. A fast viscosity solution algorithm of shape from shading for non-Lambertian surfaces[J]. Chinese Journal of Scientific Instrument, 36, 1577-1583(2015).

    [3] Horn B K P. Shape from shading: a method for obtaining the shape of a smooth opaque object from one view[D](1970).

    [4] Horn B K P[M]. Robot vision. Wang L, Jiang X L, Transl(2014).

    [5] Zhang R, Tsai P S, Cryer J E et al. Shape from shading: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21, 690-706(1999).

    [6] Durou J D, Falcone M, Sagona M. Numerical methods for shape-from-shading: a new survey with benchmarks[J]. Computer Vision and Image Understanding, 109, 22-43(2008).

    [7] Woodham R J. Photometric method for determining surface orientation from multiple images[J]. Optical Engineering, 19, 139-144(1980).

    [8] Cao Y L, Ding B J, Chen J X et al. Photometric-stereo-based defect detection system for metal parts[J]. Sensors, 22, 8374(2022).

    [9] Liu H Y, Yan Y H, Song K C et al. Efficient optical measurement of welding studs with normal maps and convolutional neural network[J]. IEEE Transactions on Instrumentation and Measurement, 70, 5000614(2021).

    [10] Bernotas G, Scorza L C T, Hansen M F et al. A photometric stereo-based 3D imaging system using computer vision and deep learning for tracking plant growth[J]. GigaScience, 8, giz056(2019).

    [11] Hao Y, Li J, Meng F et al. Photometric stereo-based depth map reconstruction for monocular capsule endoscopy[J]. Sensors, 20, 5403(2020).

    [12] Kim J, Al Faruque H, Kim S et al. Multimodal endoscopic system based on multispectral and photometric stereo imaging and analysis[J]. Biomedical Optics Express, 10, 2289-2302(2019).

    [13] Sun J A. Evaluation on an optical scanning device for skin profile measurement[J]. Journal of Cosmetic Science, 69, 9-17(2018).

    [14] Chen Z, Ji Y, Zhou M Y et al. 3D face reconstruction using color photometric stereo with uncalibrated near point lights[C](2020).

    [15] Cheng K H M, Kumar A. Contactless biometric identification using 3D finger knuckle patterns[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42, 1868-1883(2020).

    [16] Yeh C K, Matsuda N, Huang X et al. A streamlined photometric stereo framework for cultural heritage[M]. Hua G, Jégou H. Computer vision-ECCV 2016 workshops, 9913, 738-752(2016).

    [17] Jian M W, Dong J Y, Gong M G et al. Learning the traditional art of Chinese calligraphy via three-dimensional reconstruction and assessment[J]. IEEE Transactions on Multimedia, 22, 970-979(2020).

    [18] Liu W C, Wu B, Wöhler C. Effects of illumination differences on photometric stereo shape-and-albedo-from-shading for precision lunar surface reconstruction[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 136, 58-72(2018).

    [19] Peng M, Di K C, Wang Y X et al. A photogrammetric-photometric stereo method for high-resolution lunar topographic mapping using Yutu-2 rover images[J]. Remote Sensing, 13, 2975(2021).

    [21] Feng S J, Zuo C, Yin W et al. Application of deep learning technology to fringe projection 3D imaging[J]. Infrared and Laser Engineering, 49, 0303018(2020).

    [22] Wang G H, Zheng H, Zhang X. A robust checkerboard corner detection method for camera calibration based on improved YOLOX[J]. Frontiers in Physics, 9, 819019(2022).

    [23] Laga H, Jospin L V, Boussaid F et al. A survey on deep learning techniques for stereo-based depth estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44, 1738-1764(2022).

    [24] Zheng H, Wang G H, Li X C. Identifying strawberry appearance quality by vision transformers and support vector machine[J]. Journal of Food Process Engineering, 45, 14132(2022).

    [25] Zheng H, Wang G H, Li X C. YOLOX-Dense-CT: a detection algorithm for cherry tomatoes based on YOLOX and DenseNet[J]. Journal of Food Measurement and Characterization, 16, 4788-4799(2022).

    [26] Houssein E H, Mohamed R E, Ali A A. Machine learning techniques for biomedical natural language processing: a comprehensive review[J]. IEEE Access, 9, 140628-140653(2021).

    [27] Otter D W, Medina J R, Kalita J K. A survey of the usages of deep learning for natural language processing[J]. IEEE Transactions on Neural Networks and Learning Systems, 32, 604-624(2021).

    [28] Li R J, Zeng T, Peng H C et al. Deep learning segmentation of optical microscopy images improves 3-D neuron reconstruction[J]. IEEE Transactions on Medical Imaging, 36, 1533-1541(2017).

    [29] Sun Z, Wang S Y. Application of deep learning in intravascular optical coherence tomography[J]. Laser & Optoelectronics Progress, 59, 2200002(2022).

    [30] Wang F, Wang H, Bian Y M et al. Applications of deep learning in computational imaging[J]. Acta Optica Sinica, 40, 0111002(2020).

    [31] Zuo C, Feng S J, Zhang X Y et al. Deep learning based computational imaging: status, challenges, and future[J]. Acta Optica Sinica, 40, 0111003(2020).

    [32] Wang W H, Zhao X, Jiang Z X et al. Deep learning-based scattering removal of light field imaging[J]. Chinese Optics Letters, 20, 041101(2022).

    [33] Nguyen H, Novak E, Wang Z Y. Accurate 3D reconstruction via fringe-to-phase network[J]. Measurement, 190, 110663(2022).

    [34] Boukhtache S, Abdelouahab K, Berry F et al. When deep learning meets digital image correlation[J]. Optics and Lasers in Engineering, 136, 106308(2021).

    [35] Wang G H, Han J Q, Zhang X M et al. A new three-dimensional reconstruction algorithm of the lunar surface based on shape from shading method[J]. Journal of Astronautics, 30, 2265-2269(2009).

    [36] Wang G H, Han J Q, Jia H H et al. A fast viscosity solution algorithm for shape from shading[J]. Journal of Xi’an Jiaotong University, 43, 43-47(2009).

    [37] Horn B K P, Brooks M J. The variational approach to shape from shading[J]. Computer Vision, Graphics, and Image Processing, 33, 174-208(1986).

    [38] Frankot R T, Chellappa R. A method for enforcing integrability in shape from shading algorithms[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 10, 439-451(1988).

    [39] Agrawal A, Raskar R, Chellappa R. What is the range of surface reconstructions from a gradient field?[M]. Leonardis A, Bischof H, Pinz A. Computer vision-ECCV 2006, 3951, 578-591(2006).

    [40] Harker M, O’Leary P. Regularized reconstruction of a surface from its measured gradient field[J]. Journal of Mathematical Imaging and Vision, 51, 46-70(2015).

    [41] Quéau Y, Durou J D, Aujol J F. Variational methods for normal integration[J]. Journal of Mathematical Imaging and Vision, 60, 609-632(2018).

    [42] Quéau Y, Durou J D, Aujol J F. Normal integration: a survey[J]. Journal of Mathematical Imaging and Vision, 60, 576-593(2018).

    [43] Forsyth D, Ponce J[M]. Computer vision: a modern approach(2003).

    [44] Cho D, Matsushita Y, Tai Y W et al. Semi-calibrated photometric stereo[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42, 232-245(2020).

    [45] Kaya B, Kumar S, Oliveira C et al. Uncalibrated neural inverse rendering for photometric stereo of general surfaces[C], 3803-3813(2021).

    [46] Hayakawa H. Photometric stereo under a light source with arbitrary motion[J]. Journal of the Optical Society of America A, 11, 3079-3089(1994).

    [47] Belhumeur P N, Kriegman D J, Yuille A L. The bas-relief ambiguity[J]. International Journal of Computer Vision, 35, 33-44(1999).

    [48] Papadhimitri T, Favaro P. A closed-form, consistent and robust solution to uncalibrated photometric stereo via local diffuse reflectance maxima[J]. International Journal of Computer Vision, 107, 139-154(2014).

    [49] Sengupta S, Zhou H, Forkel W et al. Solving uncalibrated photometric stereo using fewer images by jointly optimizing low-rank matrix completion and integrability[J]. Journal of Mathematical Imaging and Vision, 60, 563-575(2018).

    [50] Boyd S, Parikh N, Chu E et al. Distributed optimization and statistical learning via the alternating direction method of multipliers[J]. Foundations and Trends in Machine Learning, 3, 1-122(2011).

    [51] Cho D, Matsushita Y, Tai Y W et al. Photometric stereo under non-uniform light intensities and exposures[M]. Leibe B, Matas J, Sebe N, et al. Computer vision-ECCV 2016, 9906, 170-186(2016).

    [52] Quéau Y, Wu T, Lauze F et al. A non-convex variational approach to photometric stereo under inaccurate lighting[C], 350-359(2017).

    [53] Zhang Y F, Yang Y H. Illuminant direction determination for multiple light sources[C], 269-276(2002).

    [54] Zhou W, Kambhamettu C. Estimation of illuminant direction and intensity of multiple light sources[M]. Heyden A, Sparr G, Nielsen M, et al. Computer vision-ECCV 2002, 2353, 206-220(2002).

    [55] Powell M W, Sarkar S, Goldgof D. Calibration of light sources[C], 263-269(2002).

    [56] Hara K, Nishino K, Ikeuchi K. Light source position and reflectance estimation from a single view without the distant illumination assumption[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 493-505(2005).

    [57] Park J, Sinha S N, Matsushita Y et al. Calibrating a non-isotropic near point light source using a plane[C], 2267-2274(2014).

    [58] Visentini-Scarzanella M, Kawasaki H. Simultaneous camera, light position and radiant intensity distribution calibration[C], 557-571(2015).

    [59] Quéau Y, Durix B, Wu T et al. LED-based photometric stereo: modeling, calibration and numerical solution[J]. Journal of Mathematical Imaging and Vision, 60, 313-340(2018).

    [60] Xie L M, Song Z, Jiao G H et al. A practical means for calibrating an LED-based photometric stereo system[J]. Optics and Lasers in Engineering, 64, 42-50(2015).

    [61] Liu J Y, Wang G H. Stereo vision-based calibration approach for position parameters of LED in photometric stereo[J]. Laser & Optoelectronics Progress, 59, 1615001(2022).

    [62] Shi B X, Wu Z, Mo Z P et al. A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo[C], 3707-3716(2016).

    [63] Shi B X, Mo Z P, Wu Z et al. A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41, 271-284(2019).

    [64] Mecca R, Quéau Y. Unifying diffuse and specular reflections for the photometric stereo problem[C](2016).

    [65] Mecca R, Quéau Y, Logothetis F et al. A single-lobe photometric stereo approach for heterogeneous material[J]. SIAM Journal on Imaging Sciences, 9, 1858-1888(2016).

    [66] Ju Y K. Depth learning model of non-lambert photometric stereo[D](2022).

    [69] Santo H, Samejima M, Sugano Y et al. Deep photometric stereo network[C], 501-509(2017).

    [70] Santo H, Samejima M, Sugano Y et al. Deep photometric stereo networks for determining surface normal and reflectances[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44, 114-128(2022).

    [71] Santo H, Waechter M, Matsushita Y. Deep near-light photometric stereo for spatially varying reflectances[M]. Vedaldi A, Bischof H, Brox T, et al. Computer vision-ECCV 2020, 12353, 137-152(2020).

    [72] Chen G Y, Han K, Wong K Y K. PS-FCN: a flexible learning framework for photometric stereo[M]. Ferrari V, Hebert M, Sminchisescu C, et al. Computer vision-ECCV 2018, 11213, 3-19(2018).

    [73] Chen G Y, Han K, Shi B X et al. Self-calibrating deep photometric stereo networks[C], 8731-8739(2020).

    [74] Chen G Y, Han K, Shi B X et al. Deep photometric stereo for non-Lambertian surfaces[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44, 129-142(2022).

    [75] Li J X, Robles-Kelly A, You S D et al. Learning to minify photometric stereo[C], 7560-7568(2020).

    [76] Ju Y K, Qi L, He J C et al. MPS-Net: learning to recover surface normal for multispectral photometric stereo[J]. Neurocomputing, 375, 62-70(2020).

    [77] Ju Y K, Dong J Y, Chen S. Recovering surface normal and arbitrary images: a dual regression network for photometric stereo[J]. IEEE Transactions on Image Processing, 30, 3676-3690(2021).

    [78] Ju Y K, Shi B X, Jian M W et al. NormAttention-PSN: a high-frequency region enhanced photometric stereo network with normalized attention[J]. International Journal of Computer Vision, 130, 3014-3034(2022).

    [79] Ikehata S. CNN-PS: CNN-based photometric stereo for general non-convex surfaces[M]. Ferrari V, Hebert M, Sminchisescu C, et al. Computer vision-ECCV 2018, 11219, 3-19(2018).

    [80] Zheng Q, Jia Y M, Shi B X et al. SPLINE-net: sparse photometric stereo through lighting interpolation and normal estimation networks[C], 8548-8557(2019).

    [81] Johnson M K, Adelson E H. Shape estimation in natural illumination[C], 2553-2560(2011).

    [82] Wiles O, Zisserman A. SilNet: single- and multi-view reconstruction by learning from silhouettes[C], 1-13(2017).

    [83] Curless B, Levoy M. A volumetric method for building complex models from range images[C], 303-312(1996).

    [84] Matusik W, Pfister H, Brand M et al. A data-driven reflectance model[J]. ACM Transactions on Graphics, 22, 759-769(2003).

    [86] Zheng Q, Shi B X, Pan G. Summary study of data-driven photometric stereo methods[J]. Virtual Reality & Intelligent Hardware, 2, 213-221(2020).

    [87] Alldrin N, Zickler T, Kriegman D. Photometric stereo with non-parametric and spatially-varying reflectance[C](2008).

    [88] Einarsson P, Chabert C F, Jones A et al. Relighting human locomotion with flowed reflectance fields[C], 183-194(2006).

    [89] Xiong Y, Chakrabarti A, Basri R et al. From shading to local shape[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37, 67-79(2015).

    [90] Li M, Zhou Z L, Wu Z et al. Multi-view photometric stereo: a robust solution and benchmark dataset for spatially varying isotropic materials[J]. IEEE Transactions on Image Processing, 29, 4159-4173(2020).

    [92] Logothetis F, Mecca R, Budvytis I et al. A CNN based approach for the point-light photometric stereo problem[J]. International Journal of Computer Vision, 131, 101-120(2023).

    [93] Ren J J, Wang F S, Zhang J H et al. DiLiGenT102: a photometric stereo benchmark dataset with controlled shape and material variation[C], 12571-12580(2022).

    [94] Hinton G E, Salakhutdinov R R. Reducing the dimensionality of data with neural networks[J]. Science, 313, 504-507(2006).

    [95] Hinton G E, Osindero S, Teh Y W. A fast learning algorithm for deep belief nets[J]. Neural Computation, 18, 1527-1554(2006).

    [96] Williams R J, Zipser D. A learning algorithm for continually running fully recurrent neural networks[J]. Neural Computation, 1, 270-280(1989).

    [97] LeCun Y, Bottou L, Bengio Y et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 86, 2278-2324(1998).

    [98] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[C], 1097-1105(2012).

    [100] He K M, Zhang X Y, Ren S Q et al. Deep residual learning for image recognition[C], 770-778(2016).

    [101] Huang G, Liu Z, Van Der Maaten L et al. Densely connected convolutional networks[C], 2261-2269(2017).

    [102] Iwahori Y, Woodham R J, Tanaka H et al. Neural network to reconstruct specular surface shape from its three shading images[C], 1181-1184(2002).

    [103] Rajaram K V, Parthasarathy G, Faruqi M A. A neural network approach to photometric stereo inversion of real-world reflectance maps for extracting 3-D shapes of objects[J]. IEEE Transactions on Systems, Man, and Cybernetics, 25, 1289-1300(1995).

    [104] Cheng W C. Neural-network-based photometric stereo for 3D surface reconstruction[C], 404-410(2006).

    [105] Elizondo D, Zhou S M, Chrysostomou C. Surface reconstruction techniques using neural networks to recover noisy 3D scenes[M]. Kůrková V, Neruda R, Koutník J. Artificial neural networks-ICANN 2008, 5163, 857-866(2008).

    [106] Wang X L, Fouhey D F, Gupta A. Designing deep networks for surface normal estimation[C], 539-547(2015).

    [107] Choy C B, Xu D F, Gwak J et al. 3D-R2N2: a unified approach for single and multi-view 3D object reconstruction[M]. Leibe B, Matas J, Sebe N, et al. Computer vision-ECCV 2016, 9912, 628-644(2016).

    [108] Wang N Y, Zhang Y D, Li Z W et al. Pixel2Mesh: generating 3D mesh models from single RGB images[M]. Ferrari V, Hebert M, Sminchisescu C, et al. Computer vision-ECCV 2018, 11215, 55-71(2018).

    [109] Ikehata S, Wipf D, Matsushita Y et al. Robust photometric stereo using sparse regression[C], 318-325(2012).

    [110] Nielsen J B, Jensen H W, Ramamoorthi R. On optimal, minimal BRDF sampling for reflectance acquisition[J]. ACM Transactions on Graphics, 34, 1-11(2015).

    [111] Xu Z X, Nielsen J B, Yu J Y et al. Minimal BRDF sampling for two-shot near-field reflectance acquisition[J]. ACM Transactions on Graphics, 35, 1-12(2016).

    [112] Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation[C], 640-651(2016).

    [113] Kim K, Gu J W, Tyree S et al. A lightweight approach for on-the-fly reflectance estimation[C], 20-28(2017).

    Tools

    Get Citation

    Copy Citation Text

    Guohui Wang, Yanting Lu. Application of Deep Learning Technology to Photometric Stereo Three-dimensional Reconstruction[J]. Laser & Optoelectronics Progress, 2023, 60(8): 0811011

    Download Citation

    EndNote(RIS)BibTexPlain Text
    Save article for my favorites
    Paper Information

    Category: Imaging Systems

    Received: Jan. 1, 2023

    Accepted: Feb. 22, 2023

    Published Online: Apr. 17, 2023

    The Author Email: Wang Guohui (booler@126.com)

    DOI:10.3788/LOP230431

    Topics