Journals Articles Conferences News About CLP
Submit a paper Email Alert
Search
Search
Articles
Journals
News
Advanced Search
Top Searches
2D MaterialsTransformation opticsQuantum PhotonicsSmall lasersSemiconductor UV PhotonicsSupersymmetric microring laser arrays

Acta Optica Sinica, Volume. 45, Issue 17, 1720016(2025)

Portrait Content Generation Technologies for 3D Light Field Displays (Invited)

Sheng Shen, Xinzhu Sang*, Shujun Xin, and Binbin Yan
Author Affiliations
  • School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China
    • show less
    AbstractGet PDF(in Chinese)
    Figures&Tables (14)
    Equations (0)
    References (64)
    Tools
    Paper Information
    Topics
    References(64)

    [1] Zollhöfer M, Thies J, Garrido P et al. State of the art on monocular 3D face reconstruction, tracking, and applications[J]. Computer Graphics Forum, 37, 523-550(2018).

    [2] Sharma S, Kumar V. 3D face reconstruction in deep learning era: a survey[J]. Archives of Computational Methods in Engineering, 29, 3475-3507(2022).

    [3] Kemelmacher-Shlizerman I, Basri R. 3D face reconstruction from a single image using a single reference face shape[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33, 394-405(2011).

    [4] Shen S, Xing S J, Sang X Z et al. Portrait stylized rendering for 3D light-field display based on radiation field and example guide[J]. Optics Express, 31, 29664-29675(2023).

    [5] Shen S, Xing S J, Sang X Z et al. Portrait relighting for 3D light-field display based on radiance fields[J]. Optics Communications, 572, 130920(2024).

    [6] Li N C, Yu X B, Gao X et al. Real-time representation and rendering of high-resolution 3D light field based on texture-enhanced optical flow prediction[J]. Optics Express, 32, 26478-26491(2024).

    [7] Blanz V, Vetter T. A morphable model for the synthesis of 3D faces[M]. SIGGRAPH 1999: Seminal Graphics Papers: Pushing the Boundaries, 157-164(2023).

    [8] Dou P, Shah S K, Kakadiaris I A. End-to-end 3D face reconstruction with deep neural networks[C], 5908-5917(2017).

    [9] Luo Y, Tu X G, Xie M. Learning robust 3D face reconstruction and discriminative identity representation[C], 317-321(2019).

    [10] Fan X, Cheng S C, Kang H Y et al. Dual neural networks coupling data regression with explicit priors for monocular 3D face reconstruction[J]. IEEE Transactions on Multimedia, 23, 1252-1263(2020).

    [11] Zhu W B, Wu H T, Chen Z Y et al. ReDA: reinforced differentiable attribute for 3D face reconstruction[C], 4957-4966(2020).

    [12] Mildenhall B, Srinivasan P P, Tancik M et al. NeRF: representing scenes as neural radiance fields for view synthesis[J]. Communications of the ACM, 65, 99-106(2022).

    [13] Gafni G, Thies J, Zollhöfer M et al. Dynamic neural radiance fields for monocular 4D facial avatar reconstruction[C], 8645-8654(2021).

    [14] Chan E R, Monteiro M, Kellnhofer P et al. Pi-GAN: periodic implicit generative adversarial networks for 3D-aware image synthesis[C], 5795-5805(2021).

    [15] Hong Y, Peng B, Xiao H Y et al. HeadNeRF: a realtime NeRF-based parametric head model[C], 20342-20352(2022).

    [16] Sun J X, Wang X, Zhang Y et al. FENeRF: face editing in neural radiance fields[C], 7662-7672(2022).

    [17] Du Y L, Zhang Y N, Yu H X et al. Neural radiance flow for 4D view synthesis and video processing[C], 14304-14314(2021).

    [18] Li Z Q, Niklaus S, Snavely N et al. Neural scene flow fields for space-time view synthesis of dynamic scenes[C], 6494-6504(2021).

    [19] Park K, Sinha U, Barron J T et al. Nerfies: deformable neural radiance fields[C], 5845-5854(2021).

    [20] Park K, Sinha U, Hedman P et al. HyperNeRF: a higher-dimensional representation for topologically varying neural radiance fields[EB/OL]. https:∥arxiv.org/abs/2106.13228

    [21] Pumarola A, Corona E, Pons-Moll G et al. D-NeRF: neural radiance fields for dynamic scenes[C], 10313-10322(2021).

    [22] Tretschk E, Tewari A, Golyanik V et al. Non-rigid neural radiance fields: reconstruction and novel view synthesis of a dynamic scene from monocular video[C], 12939-12950(2021).

    [23] Loper M, Mahmood N, Romero J et al. SMPL: a skinned multi-person linear model[M]. Seminal graphics papers: pushing the boundaries, 88, 851-866(2023).

    [24] Peng S D, Dong J T, Wang Q et al. Animatable neural radiance fields for modeling dynamic human bodies[C], 14294-14303(2021).

    [25] Liu L J, Habermann M, Rudnev V et al. Neural actor: neural free-view synthesis of human actors with pose control[J]. ACM Transactions on Graphics, 40, 219(2021).

    [26] Lombardi S, Simon T, Schwartz G et al. Mixture of volumetric primitives for efficient neural rendering[J]. ACM Transactions on Graphics, 40, 59(2021).

    [27] Garbin S J, Kowalski M, Estellers V et al. VolTeMorph: real-time, controllable and generalizable animation of volumetric representations[J]. Computer Graphics Forum, 43, e15117(2024).

    [28] Xu T H, Fujita Y, Matsumoto E. Surface-aligned neural radiance fields for controllable 3D human synthesis[C], 15862-15871(2022).

    [29] Thies J, Zollhöfer M, Stamminger M et al. Face2Face: real-time face capture and reenactment of RGB videos[C], 2387-2395(2016).

    [30] Zheng Y F, Abrevaya V F, Bühler M C et al. I M avatar: implicit morphable head avatars from videos[C], 13535-13545(2022).

    [31] Lei B W, Ren J Q, Feng M Y et al. A hierarchical representation network for accurate and detailed face reconstruction from in-the-wild images[C], 394-403(2023).

    [32] ZhangD B, LiuY F, LinL J et al. HRAvatar: high-quality and relightable Gaussian head avatar[C], 26285-26296(2025).

    [33] Wang Z Y, Xiang S, Deng H P et al. Panoramic three-dimensional reconstruction method based on multi-view encoded light field[J]. Laser & Optoelectronics Progress, 60, 1210009(2023).

    [34] Huang J J, Xu F, Luo L et al. Three-dimensional reconstruction of ocean waves based on mask and self-supervised learning[J]. Laser & Optoelectronics Progress, 61, 1437008(2024).

    [35] Debevec P, Hawkins T, Tchou C et al. Acquiring the reflectance field of a human face[C], 145-156(2000).

    [36] Zhao S, Jakob W, Li T M. Physics-based differentiable rendering: from theory to implementation[C], 14(2020).

    [37] Peng S D, Chen X, Zhou C X et al. Modeling emitters in indoor scenes for inverse rendering[J/OL]. Journal of Computer-Aided Design & Computer Graphics, 1-9. https:∥link.cnki.net/urlid/11.2925.tp.20240822.0747.002

    [38] Wenger A, Gardner A, Tchou C et al. Performance relighting and reflectance transformation with time-multiplexed illumination[J]. ACM Transactions on Graphics, 24, 756-764(2005).

    [39] Zhang L W, Zhang Q X, Wu M Y et al. Neural video portrait relighting in real-time via consistency modeling[C], 782-792(2021).

    [40] Zhang X M, Fanello S, Tsai Y T et al. Neural light transport for relighting and view synthesis[J]. ACM Transactions on Graphics (TOG), 40, 9(2020).

    [41] Zheng J L, Jang Y, Papaioannou A et al. ILSH: the imperial light-stage head dataset for human head view synthesis[C], 1104-1112(2023).

    [42] Choi J M, Christman M, Sengupta R. Personalized video relighting with an at-home light stage[M]. Computer vision‒ECCV 2024, 15098, 394-410(2024).

    [43] Sun T C, Barron J T, Tsai Y T et al. Single image portrait relighting[J]. ACM Transactions on Graphics, 38, 79(2019).

    [44] Shu Z X, Hadap S, Shechtman E et al. Portrait lighting transfer using a mass transport approach[J]. ACM Transactions on Graphics, 36, 2(2017).

    [45] Nestmeyer T, Lalonde J F, Matthews I et al. Learning physics-guided face relighting under directional light[C], 5123-5132(2020).

    [46] Suwajanakorn S, Seitz S M, Kemelmacher-Shlizerman I. Synthesizing Obama: learning lip sync from audio[J]. ACM Transactions on Graphics, 36, 95(2017).

    [47] Cudeiro D, Bolkart T, Laidlaw C et al. Capture, learning, and synthesis of 3D speaking styles[C], 10093-10103(2019).

    [48] Thies J, Elgharib M, Tewari A et al. Neural voice puppetry: audio-driven facial reenactment[M]. Computer vision‒ECCV 2020, 12361, 716-731(2020).

    [49] Prajwal K R, Mukhopadhyay R, Namboodiri V P et al. A lip sync expert is all you need for speech to lip generation in the wild[C], 484-492(2020).

    [50] Gao C, Shih Y, Lai W S et al. Portrait neural radiance fields from a single image[EB/OL]. https:∥arxiv.org/abs/2012.05903

    [51] Guo Y D, Chen K Y, Liang S et al. AD-NeRF: audio driven neural radiance fields for talking head synthesis[C], 5764-5774(2021).

    [52] Liu X, Xu Y H, Wu Q Y et al. Semantic-aware implicit neural audio-driven video portrait generation[M]. Computer vision‒ECCV 2022, 13697, 106-125(2022).

    [53] Shen S, Li W H, Zhu Z et al. Learning dynamic facial radiance fields for few-shot talking head synthesis[M]. Computer vision‒ECCV 2022, 13672, 666-682(2022).

    [54] Tang J, Wang K, Zhou H et al. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition[J]. International Journal of Computer Vision, 1-12(2025).

    [55] Ye Z H, Jiang Z Y, Ren Y et al. GeneFace: generalized and high-fidelity audio-driven 3D talking face synthesis[EB/OL]. https:∥arxiv.org/abs/2301.13430

    [56] Li J H, Zhang J W, Bai X et al. Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis[C], 7534-7544(2023).

    [57] Kim H, Garrido P, Tewari A et al. Deep video portraits[J]. ACM Transactions on Graphics, 37, 163(2018).

    [58] Thies J, Zollhöfer M, Nießner M. Deferred neural rendering: image synthesis using neural textures[J]. ACM Transactions on Graphics, 38, 66(2019).

    [59] Zheng M W, Yang H Y, Huang D et al. ImFace: a nonlinear 3D morphable face model with implicit neural representations[C], 20311-20320(2022).

    [60] Zheng Y F, Wang Y F, Wetzstein G et al. PointAvatar: deformable point-based head avatars from videos[C], 21057-21067(2023).

    [61] Grassal P W, Prinzler M, Leistner T et al. Neural head avatars from monocular RGB videos[C], 18653-18664(2022).

    [62] Li T Y, Bolkart T, Black M J et al. Learning a model of facial shape and expression from 4D scans[J]. ACM Transactions on Graphics, 36, 194(2017).

    [63] Zielonka W, Bolkart T, Thies J. Instant volumetric head avatars[C], 4574-4584(2023).

    [64] Müller T, Evans A, Schied C et al. Instant neural graphics primitives with a multiresolution Hash encoding[J]. ACM Transactions on Graphics, 41, 102(2022).

    Tools

    Get Citation

    Copy Citation Text

    Sheng Shen, Xinzhu Sang, Shujun Xin, Binbin Yan. Portrait Content Generation Technologies for 3D Light Field Displays (Invited)[J]. Acta Optica Sinica, 2025, 45(17): 1720016

    Download Citation

    EndNote(RIS)BibTexPlain Text
    Set citation alerts for article
    Save article for my favorites
    Paper Information

    Category: Optics in Computing

    Received: Jun. 4, 2025

    Accepted: Aug. 14, 2025

    Published Online: Sep. 3, 2025

    The Author Email: Xinzhu Sang (xzsang@bupt.edu.cn)

    DOI:10.3788/AOS251216

    CSTR:32393.14.AOS251216

    Topics

    laser devices and laser physics
    Lasers and Laser Optics
    Laser physics
    laser manufacturing
    Instrumentation, Measurement and Metrology