Laser & Optoelectronics Progress, Volume. 58, Issue 18, 1811012(2021)

Overview of 4D Light Field Representation

Yaning Li, Xue Wang, Guoqing Zhou, and Qing Wang*
Author Affiliations
  • School of Computer Science, Northwestern Polytechnical University, Xi'an, Shaanxi 710072, China
  • show less
    References(117)

    [1] Suo J L, Liu Y B, Ji X Y et al. Computational photography: keys, methods and applications[J]. Acta Automatica Sinica, 41, 669-685(2015).

    [2] Levoy M. Light fields and computational imaging[J]. Computer, 39, 46-55(2006).

    [3] Ng R. Digital light field photography[D](2006).

    [4] Sugita K, Takahashi K, Naemura T et al. Focus measurement on programmable graphics hardware for all in-focus rendering from light fields[C]. //IEEE Virtual Reality 2004, March 27-31, 2004, Chicago, IL, USA., 255-256(2004).

    [5] Kalantari N K, Ramamoorthi R. Deep high dynamic range imaging of dynamic scenes[J]. ACM Transactions on Graphics, 36, 11487-11505(2017).

    [6] Dansereau D G, Pizarro O, Williams S B. Linear volumetric focus for light field cameras[J]. ACM Transactions on Graphics, 34, 15(2015).

    [7] Viganò N, der Sarkissian H, Herzog C et al. Tomographic approach for the quantitative scene reconstruction from light field images[J]. Optics Express, 26, 22574-22602(2018).

    [9] Bouman K L, Johnson M D, Zoran D et al. Computational imaging for VLBI image reconstruction[C]. //2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 27-30, 2016, Las Vegas, NV, USA., 913-922(2016).

    [10] Raskar R, Tumblin J. Computational photography[C]. //ACM SIGGRAPH 2007 courses on-SIGGRAPH’07, August 5-9, 2007, San Diego, California.(2007).

    [12] Wu G C, Masia B, Jarabo A et al. Light field image processing: an overview[J]. IEEE Journal of Selected Topics in Signal Processing, 11, 926-954(2017).

    [13] Cai Z W, Liu X L, Pedrini G et al. Accurate depth estimation in structured light fields[J]. Optics Express, 27, 13532-13546(2019).

    [15] Zhu H, Wang Q, Yu J Y. Light field imaging: models, calibrations, reconstructions, and applications[J]. Frontiers of Information Technology & Electronic Engineering, 18, 1236-1249(2017).

    [16] Momonoi Y, Yamamoto K, Yokote Y et al. Light field mirage using multiple flat-panel light field displays[J]. Optics Express, 29, 10406-10423(2021).

    [18] Chai J X, Chan S C, Shum H Y et al. Plenoptic sampling[C]. //Proceedings of the 27th annual conference on Computer graphics and interactive techniques-SIGGRAPH’00, July 23-28, 2000, New Orleans, LA, USA., 307-318(2000).

    [19] Sakamoto T, Kodama K, Hamamoto T. A study on efficient compression of multi-focus images for denselight-field reconstruction[C]. //2012 Visual Communications and Image Processing, November 27-30, 2012, San Diego, CA, USA., 13249324(2012).

    [20] Adelson E H, Bergen J R. The plenoptic function and the elements of early vision[M]. //Landy M, Movshon J K. Computational models of visual processing(1991).

    [23] Levoy M, Hanrahan P. Light field rendering[C]. //Proceedings of the 23rd annual conference on Computer graphics and interactive techniques-SIGGRAPH’96, August 4-9, 1996, New York, NY, United States., 31-42(1996).

    [24] Zhou Z L. Research on light field imaging technology[D](2012).

    [25] Gortler S J, Grzeszczuk R, Szeliski R et al. The lumigraph[C]. //Proceedings of the 23rd annual conference on Computer graphics and interactive techniques-SIGGRAPH’96, August 4-9, 1996, New York, NY, United States., 43-54(1996).

    [26] Zhang Q, Ling J B, Wang Q et al. Ray-space projection model for light field camera[C]. //2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 15-20, 2019, Long Beach, CA, USA., 10113-10121(2019).

    [27] Zhang Q, Wang Q, Li H D et al. Ray-space epipolar geometry for light field cameras[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 5949, 1-14(2020).

    [28] Yu Z, Guo X Q, Ling H B et al. Line assisted light field triangulation and stereo matching[C]. //2013 IEEE International Conference on Computer Vision, December 1-8, 2013, Sydney, NSW, Australia., 2792-2799(2013).

    [29] Jeon H G, Park J, Choe G et al. Accurate depth map estimation from a lenslet light field camera[C]. //2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 7-12, 2015, Boston, MA, USA., 1547-1555(2015).

    [30] Wang T C, Efros A A, Ramamoorthi R. Occlusion-aware depth estimation using light-field cameras[C]. //2015 IEEE International Conference on Computer Vision (ICCV), December 7-13, 2015, Santiago, Chile., 3487-3495(2015).

    [31] Zhu H, Wang Q, Yu J Y. Occlusion-model guided antiocclusion depth estimation in light field[J]. IEEE Journal of Selected Topics in Signal Processing, 11, 965-978(2017).

    [32] Shum H Y, Chan S C, Kang S B. Image-based rendering[M](2008).

    [33] Eisemann M, de Decker B, Magnor M et al. Floating textures[J]. Computer Graphics Forum, 27, 409-418(2008).

    [34] Goesele M, Ackermann J, Fuhrmann S et al. Ambient point clouds for view interpolation[C]. //ACM SIGGRAPH 2010 papers on-SIGGRAPH’10, July 26-30, 2010, Los Angeles, California., 95(2010).

    [36] Chaurasia G, Duchene S, Sorkine-Hornung O et al. Depth synthesis and local warps for plausible image-based navigation[J]. ACM Transactions on Graphics, 32, 30(2013).

    [37] Penner E, Zhang L. Soft 3D reconstruction for view synthesis[J]. ACM Transactions on Graphics, 36, 235(2017).

    [38] Srinivasan P P, Wang T Z, Sreelal A et al. Learning to synthesize a 4D RGBD light field from a single image[C]. //2017 IEEE International Conference on Computer Vision (ICCV), October 22-29, 2017, Venice, Italy., 2262-2270(2017).

    [39] Wang T C, Zhu J Y, Kalantari N K et al. Light field video capture using a learning-based hybrid imaging system[J]. ACM Transactions on Graphics, 36, 133(2017).

    [41] Wu J. Introduction to convolutional neural networks[R], 5, 23(2017).

    [42] Wang Y L, Liu F, Wang Z L et al. End-to-end view synthesis for light field imaging with pseudo 4DCNN[M]. //Ferrari V, Hebert M, Sminchisescu C, et al. Computer vision-ECCV 2018. Lecture notes in computer science, 11206, 340-355(2018).

    [43] Yeung H W F, Hou J H, Chen J et al. Fast light field reconstruction with deep coarse-to-fine modeling of spatial-angular clues[M]. //Ferrari V, Hebert M, Sminchisescu C, et al. Computer vision-ECCV 2018. Lecture notes in computer science, 11210, 138-154(2018).

    [44] Flynn J, Neulander I, Philbin J et al. Deep stereo: learning to predict new views from the world’s imagery[C]. //2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 27-30, 2016, Las Vegas, NV, USA, 5515-5524(2016).

    [45] Kalantari N K, Wang T C, Ramamoorthi R. Learning-based view synthesis for light field cameras[J]. ACM Transactions on Graphics, 35, 193(2016).

    [46] Yoon Y, Jeon H G, Yoo D et al. Learning a deep convolutional network for light-field image super-resolution[C]. //2015 IEEE International Conference on Computer Vision Workshop (ICCVW), December 7-13, 2015, Santiago, Chile., 57-65(2015).

    [47] Zhao Y Y, Shi S X. Light-field image super-resolution based on multi-scale feature fusion[J]. Opto-Electronic Engineering, 47, 56-66(2020).

    [48] Yang F F, Li H, Peng J et al. Research on microscopic imaging of high resolution light field based on graph regularization[J]. Acta Optica Sinica, 41, 0918001(2021).

    [49] Wu J L, Guo Z H, Chen X F et al. Three-dimensional measurement method of light field imaging based on deep learning[J]. Chinese Journal of Lasers, 47, 1204005(2020).

    [50] Guo M T, Zhu H, Zhou G Q et al. Dense light field reconstruction from sparse sampling using residual network[M]. //Jawahar C V, Li H D, Mori G, et al. Computer vision-ACCV 2018. Lecture notes in computer science, 11366, 50-65(2019).

    [51] Vianello A, Ackermann J, Diebold M et al. Robust hough transform based 3D reconstruction from circular light fields[C]. //2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA., 7327-7335(2018).

    [52] Johannsen O, Sulc A, Goldluecke B. What sparse light field coding reveals about scene structure[C]. //2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 27-30, 2016, Las Vegas, NV, USA., 3262-3270(2016).

    [53] Wanner S, Goldluecke B. Globally consistent depth labeling of 4D light fields[C]. //2012 IEEE Conference on Computer Vision and Pattern Recognition, June 16-21, 2012, Providence, RI, USA., 41-48(2012).

    [54] Zhou W H, Liang L K, Zhang H et al. Scale and orientation aware EPI-patch learning for light field depth estimation[C]. //2018 24th International Conference on Pattern Recognition (ICPR), August 20-24, 2018, Beijing, China., 2362-2367(2018).

    [55] Shin C, Jeon H G, Yoon Y et al. EPINET: a fully-convolutional neural network using epipolar geometry for depth from light field images[C]. //2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA, 4748-4757(2018).

    [56] Zhu H. Spatio-angular consistency for 4D light field processing[D](2020).

    [57] Wu G C, Liu Y B, Fang L et al. Light field reconstruction using convolutional network on EPI and extended applications[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41, 1681-1694(2019).

    [58] Zhu H, Guo M T, Li H D et al. Revisiting spatio-angular trade-off in light field cameras and extended applications in super-resolution[J]. IEEE Transactions on Visualization and Computer Graphics, 27, 3019-3033(2021).

    [59] Vagharshakyan S, Bregovic R, Gotchev A. Light field reconstruction using shearlet transform[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40, 133-147(2018).

    [60] Xie S L, Wang P, Sang X Z et al. Augmented reality three-dimensional display with light field fusion[J]. Optics Express, 24, 11483-11494(2016).

    [62] Isaksen A, McMillan L, Gortler S J. Dynamically reparameterized light fields[C]. Proceedings of the 27th annual conference on Computer graphics and interactive techniques-SIGGRAPH’00, July 1, 2000, New York, USA., 297-306(2000).

    [64] Levin A, Durand F. Linear view synthesis using a dimensionality gap light field prior[C]. //2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 13-18, 2010, San Francisco, CA, USA., 1831-1838(2010).

    [65] Lin H T, Chen C, Kang S B et al. Depth recovery from light field using focal stack symmetry[C]. //2015 IEEE International Conference on Computer Vision (ICCV), December 7-13, 2015, Santiago, Chile., 3451-3459(2015).

    [66] Huang Z Y, Fessler J A, Norris T B et al. Light-field reconstruction and depth estimation from focal stack images using convolutional neural networks[C]. //ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 4-8, 2020, Barcelona, Spain., 8648-8652(2020).

    [67] Liu C, Qiu J, Jiang M. Light field reconstruction from projection modeling of focal stack[J]. Optics Express, 25, 11377-11388(2017).

    [68] Herman G T. Fundamentals of computerized tomography: image reconstruction from projections[M](2009).

    [69] Pan X C, Sidky E Y, Vannier M. Why do commercial CT scanners still employ traditional, filtered back-projection for image reconstruction?[J]. Inverse Problems, 25, 1230009(2009).

    [70] Landweber L. An iteration formula for Fredholm integral equations of the first kind[J]. American Journal of Mathematics, 73, 615-624(1951).

    [71] Valencia S A, Rodriguez-Dagnino R M. Synthesizing stereo 3D views from focus cues in monoscopic 2D images[J]. Proceedings of SPIE, 5006, 377-388(2003).

    [72] Chen Y C, Wu Y C, Liu C H et al. Depth map generation based on depth from focus[C]. //2010 International Conference on Electronic Devices, Systems and Applications, April 11-14, 2010, Kuala Lumpur, Malaysia., 59-63(2010).

    [73] Malik A S, Shim S O, Choi T S. Depth map estimation using a robust focus measure[C]. //2007 IEEE International Conference on Image Processing, September 16-October 19, 2007, San Antonio, TX, USA., 564-567(2007).

    [74] Nayar S K. Shape from focus system for rough surfaces[J]. Physics-Based Vision: Principles and Practice: Radiometry, 1, 347-360(1993).

    [75] Nayar S K, Nakagawa Y. Shape from focus[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 824-831(1994).

    [76] Strecke M, Alperovich A, Goldluecke B. Accurate depth and normal maps from occlusion-aware focal stack symmetry[C]. //2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA., 2529-2537(2017).

    [77] Rajagopalan A N, Chaudhuri S, Mudenagudi U. Depth estimation and image restoration using defocused stereo pairs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26, 1521-1525(2004).

    [78] Takahashi K, Kobayashi Y, Fujii T. From focal stack to tensor light-field display[J]. IEEE Transactions on Image Processing, 27, 4571-4584(2018).

    [79] He K, Wang X L, Wang Z W et al. Snapshot multifocal light field microscopy[J]. Optics Express, 28, 12108-12120(2020).

    [80] Liu J, Xu T, Yue W et al. Light-field moment microscopy with noise reduction[J]. Optics Express, 23, 29154-29162(2015).

    [81] Liu F L, Kuo G, Antipa N et al. Fourier DiffuserScope: single-shot 3D Fourier light field microscopy with a diffuser[J]. Optics Express, 28, 28969-28986(2020).

    [83] Ren X F, Malik J. Learning a classification model for segmentation[C]. //Proceedings Ninth IEEE International Conference on Computer Vision, October 13-16, 2003, Nice, France., 10-17(2003).

    [84] Zhu H, Zhang Q, Wang Q. 4D light field superpixel and segmentation[C]. //2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA., 6709-6717(2017).

    [85] Khan N, Zhang Q, Kasser L et al. View-consistent 4D light field superpixel segmentation[C]. //2019 IEEE/CVF International Conference on Computer Vision (ICCV), October 27-November 2, 2019, Seoul, Korea., 7810-7818(2019).

    [86] Zhu H, Sun X M, Zhang Q et al. Full view optical flow estimation leveraged from light field superpixel[J]. IEEE Transactions on Computational Imaging, 6, 12-23(2020).

    [87] Chen J, Hou J H, Ni Y et al. Accurate light field depth estimation with superpixel regularization over partially occluded regions[J]. IEEE Transactions on Image Processing, 27, 4889-4900(2018).

    [88] Su X, Maugey T, Guillemot C. Rate-distortion optimized graph-based representation for multiview images with complex camera configurations[J]. IEEE Transactions on Image Processing, 26, 2644-2655(2017).

    [89] Karypis G, Kumar V. Multilevel k-way hypergraph partitioning[J]. VLSI Design, 11, 285-300(2000).

    [90] Rizkallah M, Su X, Maugey T et al. Geometry-aware graph transforms for light field compact representation[J]. IEEE Transactions on Image Processing, 29, 602-616(2020).

    [91] Lü X, Wang X, Wang Q et al. 4D light field segmentation from light field super-pixel hypergraph representation[J]. IEEE Transactions on Visualization and Computer Graphics, 2158, 32203022(2020).

    [92] Boykov Y, Veksler O, Zabih R. Fast approximate energy minimization via graph cuts[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23, 1222-1239(2001).

    [93] Boykov Y, Kolmogorov V. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26, 1124-1137(2004).

    [94] Maugey T, Ortega A, Frossard P. Graph-based representation for multiview image geometry[J]. IEEE Transactions on Image Processing, 24, 1573-1586(2015).

    [95] Su X, Rizkallah M, Maugey T et al. Graph-based light fields representation and coding using geometry information[C]. //2017 IEEE International Conference on Image Processing (ICIP), September 17-20, 2017, Beijing, China., 4023-4027(2017).

    [96] Rizkallah M, Su X, Maugey T et al. Graph-based transforms for predictive light field compression based on super-pixels[C]. //2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 15-20, 2018, Calgary, AB, Canada., 1718-1722(2018).

    [97] Zhou T H, Tucker R, Flynn J et al. Stereo magnification: learning view synthesis using multiplane images[J]. ACM Transactions on Graphics, 37, 65(2018).

    [98] Srinivasan P P, Tucker R, Barron J T et al. Pushing the boundaries of view extrapolation with multiplane images[C]. //2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 15-20, 2019, Long Beach, CA, USA., 175-184(2019).

    [99] Mildenhall B, Srinivasan P P, Ortiz-Cayon R et al. Local light field fusion[J]. ACM Transactions on Graphics, 38, 29(2019).

    [100] Flynn J, Broxton M, Debevec P et al. DeepView: view synthesis with learned gradient descent[C]. //2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 15-20, 2019, Long Beach, CA, USA, 2362-2371(2019).

    [101] Li Q B, Kalantari N K. Synthesizing light field from a single image with variable MPI and two network fusion[J]. ACM Transactions on Graphics, 39, 229(2020).

    [102] Tucker R, Snavely N. Single-view view synthesis with multiplane images[C]. //2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 13-19, 2020, Seattle, WA, USA., 548-557(2020).

    [103] Bracewell R N. Strip integration in radio astronomy[J]. Australian Journal of Physics, 9, 198(1956).

    [104] Levoy M. Volume rendering using the Fourier projection-slice theorem[M](1992).

    [105] Ng R. Fourier slice photography[J]. ACM Transactions on Graphics, 24, 735-744(2005).

    [106] Macovski A. Medical imaging systems[M](1983).

    [107] Deans S R. The Radon transform and some of its applications[M](1983).

    [108] Malzbender T. Fourier volume rendering[J]. ACM Transactions on Graphics, 12, 233-250(1993).

    [109] Zhao Q Q, Zhang T, Zheng W B. Research on high resolution digital refocusing of light field imaging based on microlens array[J]. Laser & Optoelectronics Progress, 53, 101001(2016).

    [110] Levin A, Hasinoff S W, Green P et al. 4D frequency analysis of computational cameras for depth of field extension[J]. ACM Transactions on Graphics, 28, 1-14(2009).

    [111] Shi L X, Hassanieh H, Davis A et al. Light field reconstruction using sparsity in the continuous Fourier domain[J]. ACM Transactions on Graphics, 34, 1-13(2014).

    [112] Levoy M, Ng R, Adams A et al. Light field microscopy[J]. ACM Transactions on Graphics, 25, 924-934(2006).

    [113] Le Pendu M, Guillemot C, Smolic A. A Fourier disparity layer representation for light fields[J]. IEEE Transactions on Image Processing, 28, 5740-5753(2019).

    [114] Le Pendu M, Smolic A. High resolution light field recovery with Fourier disparity layer completion, demosaicing, and super-resolution[C]. //2020 IEEE International Conference on Computational Photography (ICCP), April 24-26, 2020, St. Louis, MO, USA., 19653339(2020).

    [115] Dib E, Le Pendu, Guillemot C. Light field compression using Fourier disparity layers[C]. //2019 IEEE International Conference on Image Processing (ICIP), September 22-25, 2019, Taipei, Taiwan, China., 3751-3755(2019).

    [116] Le Pendu M, Ozcinar C, Smolic A. Hierarchical Fourier disparity layer transmission for light field streaming[C]. //2020 IEEE International Conference on Image Processing (ICIP), October 25-28, 2020, Abu Dhabi, United Arab Emirates., 2606-2610(2020).

    Tools

    Get Citation

    Copy Citation Text

    Yaning Li, Xue Wang, Guoqing Zhou, Qing Wang. Overview of 4D Light Field Representation[J]. Laser & Optoelectronics Progress, 2021, 58(18): 1811012

    Download Citation

    EndNote(RIS)BibTexPlain Text
    Save article for my favorites
    Paper Information

    Category: Imaging Systems

    Received: May. 31, 2021

    Accepted: Jun. 15, 2021

    Published Online: Sep. 1, 2021

    The Author Email: Wang Qing (qwang@nwpu.edu.cn)

    DOI:10.3788/LOP202158.1811012

    Topics