Computer-generated holography (CGH), combining holography with computer technology, enables the dynamic reconstruction of virtual objects, and has been widely used in diverse sectors including 3D display
Opto-Electronic Advances, Volume. 7, Issue 1, 230108(2024)
Dynamic interactive bitwise meta-holography with ultra-high computational and display frame rates
Interactive holography offers unmatched levels of immersion and user engagement in the field of future display. Despite of the substantial progress has been made in dynamic meta-holography, the realization of real-time, highly smooth interactive holography remains a significant challenge due to the computational and display frame rate limitations. In this study, we introduced a dynamic interactive bitwise meta-holography with ultra-high computational and display frame rates. To our knowledge, this is the first reported practical dynamic interactive metasurface holographic system. We spatially divided the metasurface device into multiple distinct channels, each projecting a reconstructed sub-pattern. The switching states of these channels were mapped to bitwise operations on a set of bit values, which avoids complex hologram computations, enabling an ultra-high computational frame rate. Our approach achieves a computational frame rate of 800 kHz and a display frame rate of 23 kHz on a low-power Raspberry Pi computational platform. According to this methodology, we demonstrated an interactive dynamic holographic Tetris game system that allows interactive gameplay, color display, and on-the-fly hologram creation. Our technology presents an inspiration for advanced dynamic meta-holography, which is promising for a broad range of applications including advanced human-computer interaction, real-time 3D visualization, and next-generation virtual and augmented reality systems.
Introduction
Computer-generated holography (CGH), combining holography with computer technology, enables the dynamic reconstruction of virtual objects, and has been widely used in diverse sectors including 3D display
However, the reliance on devices like spatial light modulator (SLM)
Dynamic meta-holography typically employs two methods: active and multiplexed schemes. Active schemes manipulate the basic properties of the unit structure using active materials like phase change materials
The key to high fluency in holography lies in the attainment of high computational and display frame rates. Computational frame rates, referring to the speed of data computation, processing, and preparation for display, ensure that the system can calculate the required content in real time. Many current schemes depend heavily on the continuous computation of the Fast Fourier Transform (FFT) one or more times and loading it onto the SLM/DMD. This often requires specialized computing units such as GPUs
In this paper, we propose a bitwise meta-holography (Bit-MH) technology designed to achieve dynamic interactive holography with high computational and display frame rates. We spatially partition the display function of the metasurface into distinct channels with space channel multiplexing technology, each projecting a reconstructed sub-pattern. Efficient dynamic refreshing is achieved by mapping the switching states of the channels to a set of bit values and subsequently controlling the state of the channels via bitwise operations. Tests conducted on the computational platform utilized in this study resulted in a maximum computational frame rate of up to 800 kHz, and concurrently, the DMD used allowed for a maximum display frame rate of up to 23 kHz. As a proof of concept, we constructed an interactive holographic game system for the typical Tetris game in the visible region. Our design enables efficient dynamic updating of the hologram and allows interaction with external inputs. To our knowledge, this is the first demonstration of a practical interactive meta-holography system. We believe that the effective and programmable Bit-MH method can pave the way towards the realization of the seamlessly interactive holographic system.
Materials and methods
The design principle of Bit-MH
Our concept of bitwise meta-holography involves encoding and mapping the dynamic update process of the hologram as bitwise operations on a bit set. This technique permits efficient and unrestricted manipulation of the dynamic state of the hologram, circumventing any complex calculations such as Gerchberg-Saxton (G-S) iterative algorithms
In our design, the display functionality of the metasurface device is compartmentalized into multiple distinct channels, each of which projects an individual reconstructed sub-pattern, and the final displayed reconstructed image is composed of sub-patterns from all activated channels together. Dynamic refresh is achieved by selectively turning these channels on or off. The switching state of each channel is encoded as a bit value, and the collective states of all channels are consolidated into a bit set. Efficient dynamic refreshing can be achieved by performing bitwise operations on this bit set.
Since our scheme aims to control the states of multiple channels efficiently, the division of channels can be achieved via techniques such as space channel multiplexing
Figure 1.
In traditional dynamic holography, the hologram update process relies on the continuous usage of the FFT algorithm to compute the hologram that’s loaded onto a dynamic refresh device such as a DMD. This method necessitates the computation of the entire image each time, resulting in a time complexity of up to
Our approach segregates the hologram into
Figure 2.
To compare the computational time complexities of two algorithms directly, we executed benchmark measurements on this demonstration platform focusing on computational frame rate generation speed with 600 × 600 pixels image. The computational frame rate generation speed of our method can reach up to 800 kHz, while the single Fourier transform can achieve approximately 30 Hz. On our computing platform, our methodology can enhance the computational frame rate by more than about
The design and setup of the interactive Tetris holographic game system
For the purpose of demonstrating Bit-MH actually, we designed and constructed a polarization-insensitive dielectric metasurface device that operates in transmission mode within the visible light band for an interactive Tetris game system.
We chose a silicon nitride nanopillar with a circular cross-section with different radii and 700 nm height. Each nanopillar is located on a silica square substrate of 440 nm period and is illustrated in
Figure 3.
Finally, we fabricated our metasurface device according to the designed geometries of nanopillars using electron beam lithography (EBL). The optical image of the processed device characterized by the confocal laser scanning microscope (CLSM) is shown in
To satisfy the dynamic and rich display requirements of the interactive game, we have carefully designed and divided the functions of the different channels. The
We have designed 144 channel regions with varying functions for the Tetris game (The size of the hologram was 3000 rows and 3000 columns of pixels). The division and arrangement of the channels with diverse shapes and size proportions across the entire device are illustrated in
Figure 4.
In our designed Tetris game, the No. 24 channel was a static channel displaying the main game scene with static borders and texts (S=1); channels No. 25 to 33 were cyclic channels that switched and refreshed at a fixed frequency to display a rotating cube (C=9); the rest of the 134 channels were multiplexed channels (M=134). Therefore, the design can achieve theoretical frame number
We utilize the DMD to produce the structured light field, governing the opening states of channel regions, thus there’s no limitation to channel’s shape. For simplicity, while preserving the study’s generality, we opted for a predominantly rectangular channel shape.
The control program running on the low-power embedded platform (Raspberry Pi 4b, 64-bit Raspbian operating system) was developed using C++ programming language, detailed software design and development contents are described Supplementary information Section 4. A regular gamepad (Microsoft Xbox wireless controller) was employed in communicating with the micro-controller (Raspberry Pi 4b, Raspberry Pi Foundation, UK) via a standard Bluetooth communication interface.
In the game process, when a control signal from the gamepad was received, the button identifier was decoded and obtained by the microcontroller. This identifier was then mapped to game action, such as block movement or rotation. Subsequent to the execution of the control operation, the current game states were verified, and the score and level information were updated. All necessary information was immediately encoded into the bit set of game states.
The DMD control module was concurrently running in the background, where the channel states were read, and the corresponding compact binary control pattern was rendered. This pattern was then written to the standard HDMI output terminal of the microcontroller for the transmission of control data to the DMD, affecting the modulation of the incident light field. This process was continuously executed in a high-speed loop, ensuring constant response to user input. Benefiting from our bitwise dynamic meta-holography technology, game systems that require high computational frame rates and display frames could be operated even on a low-power computing platform.
Results and discussion
We further demonstrate our interactive holographic game system based on Bit-MH through an actual experimental system. To our knowledge, this is the first demonstration of a practical interactive meta-holography system. The system's basic components were, in order, the laser light source, the DMD, the metasurface device, and the requisite optical components. The schematic diagram of the optical system can be found in Supplementary information Section 5 (in
For a more straightforward illustration, we began by freely manipulating the on/off states of channels in debug mode to display a static pattern composed of independent or combined reconstructed sub-patterns from different channels. A 532 nm laser was employed as the light source. We added dark gaps around each channel through the control program to suppress inter-channel crosstalk.
We activated a single channel indicating clearance fireworks to exhibit an independent sub-patterns, as shown in
Figure 5.
In
Furthermore, our scheme can enable color display and the real-time combination of basic sub-patterns. The capability of the Bit-MH allowed us to efficiently manipulate different channels simultaneously, which, in addition to the interactive game demonstrated earlier, allowed for the flexible combination of different sub-patterns to improvise unplanned patterns.
To exemplify this, we used the array of squares in the main game scene area to construct more complex patterns composed of basic squares independent of the game and without predefinition. In the debug mode of our control program, we activated the corresponding channels based on the content of the temporarily conceived pattern, immediately generating the desired pattern. We displayed a blue uppercase text ‘I’ text pattern at 473 nm, a red heart pattern at 633 nm, and a “HUST” text pattern of the school logo in green at 532 nm, as shown in
Finally, we employed a power meter to measure the optical power impinging upon the entire metasurface (
Experimental results validated that our bitwise dynamic meta-holography scheme offers the high computational and display frame rates required by interactive holographic systems, for instance, and retains substantial manipulation flexibility.
Conclusions
In this paper, we introduced a dynamic interactive bitwise meta-holography that enables dynamic holographic displays with ultra-high computational and display frame rates. The metasurface device was spatially divided into multiple different channel regions, each of which projects a reconstructed sub-pattern. The process of updating the hologram was mapped to bitwise operations on a set of bit values, thus avoiding any complex hologram computation during dynamic refresh. These bitwise operations not only enable efficient computation, but also facilitate the quick and simple construction of update masks for on-demand alterations, thereby significantly reducing computational demands. Our methodology requires very low computational power and can be executed even on low-power computing devices at high speed. On the Raspberry Pi computational platform as demonstrated, we measured a maximum computational frame rate of up to 800 kHz, and the DMD provided a maximum display frame rate of up to 23 kHz.
Furthermore, modern CPUs provide vectorized instructions, such as Single Instruction and Multiple Data (SIMD), which enable simultaneous operation on many bit values. Owing to the partitioned control and simplicity of image structure, our strategy can effectively utilize vectorized instructions and parallel programming techniques to further boost the efficiency of bitwise operations and image generation.
The unmodulated zero-order light observed in the experimental results can be eliminating through filtering, tailored device design, and refining hologram computation algorithms. Implementing these measures can significantly boost efficiency and improve the quality of the hologram. Detailed methods for eliminating zero-order light are described in the Supplementary information Section 6.
While the overall device in this study was primarily designed for the Tetris game, by strategically planning the basic meta-display patterns, our scheme has the potential to generate many complexes and undefined patterns on the fly with flexible and efficient dynamic control, promising candidate for building a general-purpose dynamic meta-holography system. Notably, our system allows for expanding more prosperous interaction methods, such as gesture control
Our proposed dynamic interactive bitwise meta-holography can inspire the related field, offering a practical and powerful approach for advanced human-computer interaction, real-time 3D visualization, and next-generation virtual and augmented reality systems, etc.
[32] RW Gerchberg. A practical algorithm for the determination of phase from image and diffraction plane pictures. Optik, 35, 237-246(1972).
Get Citation
Copy Citation Text
Yuncheng Liu, Ke Xu, Xuhao Fan, Xinger Wang, Xuan Yu, Wei Xiong, Hui Gao. Dynamic interactive bitwise meta-holography with ultra-high computational and display frame rates[J]. Opto-Electronic Advances, 2024, 7(1): 230108
Category: Research Articles
Received: Jul. 7, 2023
Accepted: Sep. 27, 2023
Published Online: Apr. 19, 2024
The Author Email: Xiong Wei (WXiong), Gao Hui (HGao)