Three-dimensional (3D) display is one of the most promising displays that provide realistic 3D images and has the potential to revolutionize consumer electronics used in entertainment, education, healthcare, manufacturing, and beyond^1-4. Tabletop 3D display provides a brand-new display form that supports multi-user collaborative works and interaction^5-7, which is expected to redefine the traditional wall and portable display forms.

To obtain a good 3D tabletop viewing experience, the following requirements must be satisfied. Firstly, the 3D viewing angle in the radial direction should be large enough so that viewers can properly see 3D objects from large oblique viewing positions around the table. Since the 3D viewing angle of 360° in the circumferential direction is the most basic requirement for the tabletop 3D displays, the viewing angle in this direction can be ignored. Secondly, correct perspective and parallax in the radial direction should be provided so that the image perspective changes correctly as viewers move forward and backward. Thirdly, the number of viewers should not be limited. In other words, the 3D images should be seen by multiple viewers simultaneously, no matter how many viewers in the space. Finally, the viewers should not wear special glasses or headsets. And the tracking scheme should not be adopted because it will limit the number of viewers^{8, 9}.

The above four requirements should be satisfied at the same time. Among them, the first and second requirements are the most important. Holographic display^10-15 and volumetric display^16-18 have been used to realize the tabletop 3D displays. They satisfy all the requirements except the first important one. Importantly, they are not the ideal choice for the tabletop 3D display regarding color, size, and transparency of 3D images. Light field display^19-26 based on projector arrays has been used to realize the tabletop 3D display recently. The light field display not only addresses the critical visual fatigue issues in the conventional 3D displays but also has multiple advantages including large display size, correct occlusion relationship, full-color 3D images, and so on. It satisfies the first, third, and last requirements above, but remains a challenge for the second requirement of the radial perspective and parallax due to the limited amount of spatial information unless a tracking system is used²⁷. However, the last requirement not to track cannot be satisfied in this case.

Integral imaging is a kind of light field display technology that provides an improved solution for the tabletop 3D display, and it can satisfy the last three conditions simultaneously due to several advantages in principle^28-32. Integral imaging captures and reproduces a light field by using a lens array. In the capture process, a 3D scene is captured into an elemental image array (EIA) through the lens array. In the reproduction process, the EIA is reproduced into a 3D image through the lens array. Both circumferential and radial parallaxes are provided, and full-color 3D images with correct occlusion relationships can be viewed by multiple viewers^33-34. No special glasses or headsets are needed. However, the first requirement of the large radial 3D viewing angle is still challenging at present, especially in the case of a large display size. By designing special lenses such as conical lenses³⁵ to be the lens units in previous works, the 3D viewing angle range in the radial direction can be increased and the shape of the viewing area can be adjusted^36-40. In terms of the cone viewing area, the 3D viewing angle range can be up to −30° to 30° by using triplet lenses in the case of a display size of 23.6 inches³⁹. In terms of the ring-shaped viewing area, the effective 3D viewing angle range from the inner ring to the outer ring can achieve ±20° to ±35° in the display size of 27 inches⁴⁰. Optimizing lenses provides a pathway for the improvement of 3D viewing angle in the radial direction. However, since the 3D viewing angle is an intersection angle of all the lens units’ fields of view, it is difficult to extend the 3D viewing angle in the radial direction when the display size significantly increases.

In this paper, we propose an integral imaging-based tabletop light field 3D display with a large radial 3D viewing angle in the case of a large display size. A compound lens array is designed to greatly extend the radial 3D viewing angle in the case of a limited amount of spatial information while balancing other important 3D display parameters. A new light field capture model is developed to guide the generation of EIAs based on backward ray-tracing technology. Additionally, a new method for correcting the distortion of 3D images is proposed to obtain high-quality 3D images. A 43.5-inch prototype is developed, and the radial 3D viewing angle ranges from −34.4° to 34.3°, which is larger than the conventional tabletop 3D displays. High-resolution 3D images can be observed at large radial 3D viewing positions.

The proposed tabletop light field 3D display for producing the 3D images in a larger viewing angle range is schematically shown in Fig. 1(a). The bottom of the tabletop light field 3D display is two spliced 31.5-inch 8K off-the-shelf purchased liquid crystal display (LCD) panels, and they are the refreshable image generator. The aspect ratio of each 8K LCD panel is 16:9. Splicing of two panels makes the aspect ratio of the total tabletop light field 3D display reaches around 16:18, which is closer to a square. Thus, different viewers around the table are approximately the same distance from the center of the tabletop light field 3D display, resulting in the same radial perspective. The display size is 43.5 inches.

The 8K LCD panels display the EIA which consists of multiple periodically arranged elemental images. Each elemental image corresponds to a compound lens unit above it. Divergent LED light uniformly illuminates the pixels of the LCD panels. The emergent beam from the pixels is modulated by the corresponding compound lens unit, and sharp spots are then imaged on the plane of the light shaping diffuser screen. As shown in Fig. 1(b), through the modulation of multiple compound lens units, multiple sharp spots related to the same 3D object points are overlapped to be voxels which are the basic 3D display units of the tabletop light field 3D display. The voxels are located within the depth range above and below the light shaping diffuser screen. Thanks to the designed three-piece compound lens unit with aberration correction, the edge pixels of the elemental image can be imaged as the smallest blur spots in a large field of view as large as 70°. It makes the voxels as clear as possible. Therefore, the viewers can view high-definition 3D images from a large radial viewing angle.

As for the light shaping diffuser screen, it diffuses the light emerging from the voxels, thereby resolving the gap issue between adjacent lens units. The diffusing angles in the horizontal and radial directions are identical due to the full-parallax property of integral imaging. We choose off-the-shelf diffusers because they are convenient and inexpensive.

The overall 3D viewing angle of the tabletop light field 3D display is the overlapped angle of all the lens units’ viewing angles. In the proposed tabletop light field 3D display, the viewing angles of all lens units are converged to the center to ensure a maximum overlapped viewing angle. Since each designed compound lens has an enhanced field of view with good image quality, the overall 3D viewing angle in the radial direction is increased. In addition, the diffusion effect of the light shaping diffuser screen makes the 3D viewing angle exceed σ, where σ denotes the diffusing angle. The overall 3D viewing angle in the radial direction can be denoted as

where p is the pitch of the compound lens array, g is the distance between the planes of LCD panel and compound lens array, and D_CL is the distance between the viewpoints and compound lens array planes.

The tabletop light field 3D display needs a large 3D viewing angle in the radial direction to ensure that correct and high-quality 3D images can be seen from large oblique viewing positions. However, the design of the lens array for the large 3D viewing angle is challenging. The reasons are as follows.

Firstly, the 3D viewing angle is determined by the pitch p of the lens array and the gap g between the lens array and the LCD panel, as depicted in equation (1). Larger pitch p and smaller gap g can improve the 3D viewing angle. In other words, lenses with a large relative aperture are required to improve the 3D viewing angle. However, the other two important 3D display parameters of spatial resolution and depth of field will significantly deteriorate in this case. The relationship between the spatial resolution, the depth of field, and the 3D viewing angle is expressed as

where R_I and R_d represent the spatial resolutions of the 3D image and the LCD panel, respectively, and Z_depth represents the depth of field.

Secondly, the whole size of the lens array for display purposes is about 43.5 inches. In consideration of the weight, flatness, cost, and alignment difficulty in the fabrication of this large-size lens array, increasing the number of lenses in a lens unit for correcting aberrations does not always work. Product reliability and implementation difficulty should also be considered.

Here, we design a compound lens array composed of three-piece compound lens units. The structure of the compound lens array is shown in Fig. 2(a). From a practical point of view, each compound lens unit consists of three simple spherical lenses arranged along a common axis with different materials and different surfaces. Through balancing all the principal 3D parameters, including the spatial resolution, the 3D viewing angle, and the depth of field, the pitch p of the compound lens array is set to 13 mm, the entrance pupil diameter of the lens unit is set to 8 mm, and the focal length is set to 11 mm. The balance of all the principal 3D parameters is mainly based on the constraints between these parameters shown in Eq. (1) and the minimum requirements of parameters for optimal viewing characteristics. The pitch p is set to 13 mm to ensure a super multi-viewpoint, an improved viewing angle, an appropriate depth of field and a proper spatial resolution. The entrance pupil diameter is set to 8 mm to ensure a wide beam of incident light in the pitch of 13 mm, and the focal length is set to 11 mm to ensure an improved viewing angle and a proper lens magnification. A reverse design method is employed in the design of the compound lens array. The central depth plane of the tabletop 3D display is used as the object plane, and the plane of the elemental images is used as the image plane. An initial structure is selected and then optimized after several rounds of iteration and adjustment, and the final satisfactory structure, including the front and section views, is shown in Fig. 2(b). Note that the marginal field is provided a higher weight than the central field during optimization to be consistent with the human viewing habits in the tabletop display.

Figure 2(c) shows the spot diagram of the designed compound lens. The root mean square (RMS) spot radii of all sampled fields (0°, 10.5°, 17.5°, 24.75°, and 35°) are close to the size of two pixels, one of which is the smallest luminous unit. These RMS spot radii are superior to the conventional compound lens array, and the field of view is larger. As the field angle increases from 0° to 24.75°, the RMS spot radii gradually decrease, which meets the human viewing habits of viewing 3D images obliquely. When the field angle then increases from 24.75° to 35°, the RMS spot radius increases back to 216.573 μm due to the degraded imaging quality in the marginal field. A certain amount of light cannot reach the imaging plane at 35°, resulting in some vignetting which reduces the brightness of the 3D images but does not affect the reconstruction of the 3D images.

The 3D videos displayed on the proposed tabletop light field 3D display need to be made by capturing the light field. We present a new light field capture model for the proposed tabletop light field 3D display, in which a parallelogram-shaped plenoptic map is formed, and a backward ray tracing is applied to capture the sampled plenoptic field in only one step without pixel redundancy and computing burden.

For capturing the light field, the capture system is inverse to the display. It can be schematized through a simple system in which a pinhole array is set parallel to an image sensor like a CCD. In other words, the compound lens array is simplified to a pinhole array, so we only consider rays passing through the center of the lens. In the scheme of Fig. 3(a), any pinholes in the array produce a pinhole image on the image sensor but from a different perspective. These perspective images are referred to as the elemental images, and the collection of the elemental images is referred to as the EIA.

According to the viewpoint capture theory, multiple cameras are used to simulate the viewers’ eyes by capturing the 3D scenes from viewpoints rather than lenses. One camera is shown in Fig. 3(a). The elemental images are not taken directly but are coded from sub-images taken by multiple cameras. We assume the distance between the viewpoint and pinhole planes is D'_CL, which is equal to the distance D_CL in the display process. The cameras are distributed as the viewpoint distribution, and each camera has an off-axis perspective projection.

This viewpoint-based system captures a sampled version of the plenoptic field. The plenoptic map at the plane of the image sensor can be represented by a 4D plenoptic function L(x, y,θ, φ) parametrizing the pinholes (x, y) on the pinhole plane and the angular inclination (θ, φ) of the rays²⁸. This 4D plenoptic function is simplified to a two-dimensional (2D) slice, that is, L(x, θ) for fixed y and φ to make the representations understandable.

Figure 3(b) shows the plenoptic map. In contrast to the rectangular-shaped plenoptic map in the conventional light field capture model^{28, 41}, a parallelogram-shaped plenoptic map is formed. In this sense, the sub-images captured from the cameras can be represented by inclined pixel sets with a slope of 1/D'_CL, and one of them is marked with green in Fig. 3(b). g' represents the distance between the image sensor and the pinhole plane. It is also interesting that the column lines correspond to the elemental images. The elemental image is computed by extracting from pixels with the same x-coordinates of all the sub-images. While in the plenoptic map, the sampling period along the spatial direction is the pitch p of the pinhole array, and the period along the angular direction is given by p_θ = p_pixel/g', where p_pixel is the pixel size of the image sensor. The gray dots in the plenoptic map indicate a set of missing rays due to the physical seam between two spliced LCD panels. This missing sampling range is determined by the width d of the seam and the position offset between the seam and the corresponding pinholes.

From another perspective, the 4D plenoptic function can also be represented by L'(m, n,x, y) parametrizing the cameras (m, n) on the viewpoint plane and the pinholes (x, y) on the pinhole plane. For a camera (m₀, n₀), its corresponding sampled plenoptic component is given by

where I and J are the numbers of the pinholes in the horizontal and vertical directions, respectively. It is apparent that for one camera, only the rays passing through the pinholes are useful for the pixels on the image sensor due to the sampling period p from the plenoptic map. However, the resolution of the images taken by the cameras is generally very high. More than 95% of pixels do not contribute to the EIA, so they are redundant. Therefore, the backward ray-tracing technology⁴² is used to capture the sampled plenoptic field simply. Rays having the same number of pixels on the image sensor are created from the viewpoint position. The direction of the rays is determined by the pinhole position (i_p, j_p) and the viewpoint position (m₀, n₀), and it is given by

Each viewpoint only shoots I × J rays which are identical to the number of pinholes. Note that the rays pointing to the physical seam do not need to be created because there are no pixels on the seam. In the ray-tracing process, the created ray may hit an object along its propagation, as shown in Fig. 3(c). We follow the ray until it finds a light source. The exact coloring and shading of the closest hit point are then figured out for assigning them the corresponding pixels on the image sensor. Each ray has its dependent thread in the tracing process. As can be seen, using backward ray tracing, the sampled plenoptic field can be captured in one step without taking multiple high-resolution sub-images in advance and without performing pixel mapping from the sub-images to the elemental images. Therefore, the efficiency of the light field capturing is improved.

In the display, the reconstructed 3D images have spatial distortions, including geometric and barrel or pincushion distortions. The geometric distortions are mainly due to the rotational and skew misalignments between the compound lens array and the spliced LCD panels, the positioning errors of the installation between two LCD panels, the inter-lens position misalignment, and the erroneous placement of other optical components. The barrel or pincushion distortions are mainly caused by the lens aberration. Since the light shaping diffuser screen used in the display has the ability to correct the barrel or pincushion distortions by changing the aperture stop⁴³, the correction of the barrel or pincushion distortions can be ignored. Here we focus on the rectification of the geometric distortions. A novel distortion correction method is presented to rectify the geometric distortions, which is carried out while checking whether the reconstructed 3D image and the reference sheet match exactly. The method is divided into two steps as follows.

Firstly, the reconstructed 3D image is corrected roughly by performing projective transformation to two sub-EIAs, which are split by the whole EIA and correspond to 3D sub-images 1 and 2, respectively. A real-world square pattern sheet with the correct shape is placed on the light shaping diffuser screen to be the reference sheet. Then, a distorted 3D image of the same square image model is displayed on the light shaping diffuser screen through the display. A camera is placed at the top of the display to capture both the reconstructed 3D image and the reference sheet. To match the distorted 3D image and the reference sheet roughly, each sub-EIA is transformed, and the example is shown in Fig. 4(a). The equation can be expressed as

where x and y denote the pixel coordinates of the tth original sub-EIA, and x' and y' denote the pixel coordinates of the tth target sub-EIA, where t = 0, 1. s is a scale factor. H_t is a 3 × 3 homography matrix. It can be calculated by four corner pairs with detected coordinates on two captured images. h₁, h₂, and h₃ stand for the three columns of the homography matrix H_t.

Next, the distorted 3D images corresponding to each compound lens are corrected individually for precise rectification. One example is shown in Fig. 4(b). Each crosshair pattern in the elemental image is imaged at the light shaping diffuser screen through the corresponding compound lens unit, then it is transformed to match the ideal printed crosshair pattern sheet placed at the light shaping diffuser screen by using an interactive feedback program. The transformation equation is given by

where i and j denote the index of the elemental images, x and y denote the pixel coordinates of the (i, j)th original elemental image that is split by the EIA after rough correction, and x' and y' denote the pixel coordinates of the (i, j)th target elemental image. H'_{i, j} is a 3×3 homography matrix, and h'₁, h'₂ and h'₃ are the three columns of H'_{i, j}.

The combination of the rough and precise correction algorithms ensures high precision in correcting geometric distortions of the 3D images. Note that the correction is performed for the 3D images on the central depth plane. 3D images on other depth planes are not be corrected separately because the geometric distortions on the central depth plane are representative in the whole tabletop light field 3D display.

We built a prototype of the proposed tabletop light field 3D display and demonstrated its 3D display performance. Figure 5(a) and 5(b) show the appearance of the prototype. Two 31.5-inch off-the-shelf LCD panels (LM315QU1-SSA1, LG Display) are used in a vertical arrangement. Each LCD panel has 7680 × 4320 pixels, and the pixels are in a vertical stripe arrangement. Considering the 3D image would lose a lot of brightness compared to the 2D image displayed on the LCD panel, and the off-the-shelf LCD panels have a low brightness of 400 cd/m², we optimized the LED backlight to increase the maximum brightness of each LCD panel to 1700 cd/m². Thus, the brightness of the 3D image is improved.

Regarding the compound lens array, the arrangement of the compound lens units is hexagonal because hexagonal grids have a higher fill factor and allow more lenses compared to rectangular grids. The hexagonal lens array also corresponds to a hexagonal pyramid viewing area, which is closer to the ideal cone viewing characteristics of the tabletop display. The total number of compound lens units in the lens array is 3751. The distance from the compound lens array plane to the light shaping diffuser screen is set to 149 mm. For the light shaping diffuser screen, the diffusing angles in the horizontal and radial directions are both 5°.

The EIA with a resolution of 7680 × 8640 pixels is rendered and transformed using a PC containing a 3.8 GHz Intel Core i7-10700K CPU, 64GB of RAM, and an NVIDIA GeForce RTX 3080 GPU. One graphics card drives two 8K LCD panels through two DP1.4 interfaces individually. In this case, the two 8K LCD panels can be synchronized without any additional hardware.

3D images in the circumferential direction are shown in Fig. 5(c) and the recorded video of Video 1 (See Supplementary information). Clear 3D images and perspectives can be viewed when the positions of the camera are rotated from 0° to 360°. Therefore, the circumferential viewing angle is 360°, which provides people around the table with a shared viewing experience. From Fig. 5(d) and the recorded video in Video 2 (See Supplementary information), correct radial parallax can be viewed. The 3D viewing angle range in the radial direction is −34.4° to 34.3°, which is larger than the conventional tabletop light field 3D display. The total 3D viewing angle in the radial direction is 68.7°. It is slightly less than the design value of 70° due to the measurement error of the viewing angle and the processing errors of the compound lens array.

We used a 1951 United States Air Force (USAF) resolution test chart to be the object to demonstrate the resolution of the 3D images viewed at a large radial viewing position of 30°. The 3D images of the resolution test chart are obtained at different viewing positions in the circumferential direction, as shown in Fig. 6(a) and 6(b). Several stitching artifacts can be seen in the USAF chart images due to the physical seam. Patterns in the red dashed boxes represent the minimum patterns that can be resolved by the eyes. The results show clear resolutions of Element 6, Group −2 at 0°, 40°, and 90°, as well as Element 1, Group −1 at 130°, 180°, 220°, 270°, and 310°. Different resolutions at different viewing positions are caused by viewing position errors in the radial direction. As can be seen in these USAF chart images, the proposed tabletop light field 3D display is able to reproduce high-quality 3D images at large radial viewing positions.

Figure 7 shows a USAF chart 3D image captured at the position directly above the tabletop light field 3D display and the zoomed-in image. The radial and circumferential viewing positions are both at 0°. The results of the resolution chart show a clear resolution of Element 4 and Group −2, which is lower than that of the large radial viewing position shown in Fig. 6(a). The performance matches the design of the compound lens array that provides a higher weight to the marginal field than to the central field. As a result, the proposed tabletop light field 3D display does provide a good solution to improve the radial 3D viewing angle.

We demonstrated an integral imaging-based tabletop light field 3D display with a large radial viewing angle in the case of a large display size. As verified by the experimental results, correct perspective and parallax can be achieved within 0°–360° in the circumferential direction and −34.4° to 34.3° in the radial direction. The display size is 43.5 inches. High-resolution 3D images can be reproduced at large radial viewing positions. In spite of the design of the compound lens array improving the radial viewing angle, there is still an upper limit due to the tradeoff between the viewing angle and other effective parameters, such as the number of viewpoints, the depth of field, and the resolution of 3D images. We can probably improve the radial viewing angle without sacrificing other parameters by increasing the total amount of spatial information (called the spatial bandwidth product). The use of the time-division multiplexing of directional backlights might improve the spatial bandwidth product, and the required hardware, including 8K to 16K 2D display panels with a high refresh rate and a high pixel density, needs to be studied in the future.

Note that the splicing of the LCD panels in the proposed display is a spatial multiplexing method to increase the spatial bandwidth product, but it is focused on increasing the display size rather than the viewing angle. As for the seam of the proposed display, in the future, a high-pixel-density LCD panel with a resolution of more than 16K may be able to achieve a better tabletop light field 3D display effect without splicing multiple LCD panels together.

We expect that this approach could be especially beneficial for glasses-free 3D displays. As for practical applications, our technology offers a 3D sharing viewing experience, which is much demanded in electronic sand tables and collaborative works. The input 3D video is compatible with various formats, including real-world light field video capture, 3D rendering in computer graphics, and 2D to 3D. Hence, this approach is expected to be integrated with the real-time capture and display systems as well as the real-time 3D interaction systems.

In summary, we proposed a tabletop light field 3D display based on integral imaging with large viewing angle and simple flat-panel configuration characteristics. A prototype was built with a large display size of 43.5 inches and a large radial viewing angle of 68.7°. Correct perspective and parallax are realized in both the circumferential and radial directions. Importantly, the increased radial viewing angle does not come at the expense of the resolution of the 3D images in large viewing positions. We anticipate that further increases in radial viewing angle will be attained by cooperating with time/spatial multiplexing methods, thereby bringing us ever closer to the goal of practical applications such as electronic sand tables and tabletop 3D games. We expect that the proposed tabletop light field 3D display has the potential to become a practical, revolutionary alternative to the traditional 3D displays.