Single-shot high-speed mapping photography is indispensable for observing transient physical, chemical, and biological processes during their occurrence, offering critical insights into the underlying mechanisms governing these events^[1–4]. This approach assigns time-gated frames of a dynamic scene to separate spatial positions so that they can be captured by one or more two-dimensional (2D) detectors. It effectively circumvents the limitations of high-speed burst charge-coupled device (CCD) sensors and cameras^[5] and computational high-speed imaging techniques such as compressive multi-aperture complementary metal-oxide-semiconductor (CMOS) image sensors and compressed optical-streaking cameras^[6–8], which often struggle with challenges related to the low fill factor of image sensors and/or the complexity of sophisticated image reconstruction algorithms. Single-shot high-speed mapping photography can be classified into the categories of active illumination^[9] and passive detection^[10]. Although the capabilities of these two approaches can often be highly complementary, both also suffer from high technical requirements. Active illumination-based methods typically require complex system configurations to transfer temporal information to other domains, such as the optical spectrum^[11,12], illumination angle^[13], spatial position^[14,15], and/or spatial frequency^[16,17] either at the illumination source or receiving sensor. Consequently, the elimination of specialized illumination is an attractive feature of passive detection-based approaches that use receive-only ultrafast detectors^[18]. To this end, the passive implementation of time-gating and image mapping is often achieved through the introduction of multiple sensors and/or active optical elements, with examples including rotating steering mirrors combined with a CCD camera array^[19], multiple beam splitters combined with intensified CCD cameras^[20], and the use of image-convertor streak tubes^[21]. However, the introduction of multiple sensors and/or active elements often leads to passive detection-based methods to face challenges in light throughput and cost efficiency.

Among recent advancements, a noticeable development is diffraction-gated real-time ultrahigh-speed mapping (DRUM) photography^[22]. Using the inter-pattern transition of a digital micromirror device (DMD)^[23–27], DRUM photography leverages a sweeping optical diffraction envelope as a nanosecond-level time gate to sequentially extract frames at different time points from distinct diffraction orders. The system can capture a transient event in a single exposure at 4.8 million frames per second (Mfps) and a sequence depth of 7 frames.

Even with these attractive features, the technical specifications of DRUM photography have the potential for improvement. Currently, DRUM photography only records diffraction orders distributed along a single horizontal axis, thus limiting its sequence depth and imaging speed. More significantly, only transitions between uniform DMD patterns (i.e., “all-on” or “all-off”) have so far been employed for diffraction gating in DRUM photography, a restriction that imposes an intensity-only image contrast while also overlooking the DMD’s ability to serve as a programmable spatial filter^[28,29].

To overcome these limitations, we develop DRUM schlieren (DRUMS) photography. First, by leveraging information from both on-axis and off-axis diffraction orders, DRUMS photography doubles the sequence depth and imaging speed of DRUM photography. Second, by recognizing the DMD as a dual-purpose device for both nanosecond-level diffraction gating and Fourier plane spatial filtering, DRUMS photography introduces schlieren imaging^[30–32] capabilities to the DRUM photography platform for ultrahigh-speed image acquisition. Benefiting from this ultrahigh-speed schlieren imaging ability, we apply DRUMS photography to the real-time observation of laser-induced breakdown in distilled water.

The DRUMS photography system is schematically illustrated in Fig. 1(a). A 473 nm continuous-wave laser (CNI, MSL-FN-473) serves as the illumination source for probing a transient scene. After passing through front optics chosen to suit the desired field of view (FOV), the transmitted light is reflected by a beam splitter (Thorlabs, BP250) to form an image of the transient scene on the intermediate image plane (IIP). The image is processed by a folded 4f imaging system, consisting of a stereoscopic objective lens (Olympus, MVPLAPO2XC, 0.5 NA) and a DMD (Ajile Light Industries, AJD-4500). The DMD features an array of 912×1140 micromirrors of horizontal pitch p=10.8μm and width w=7.3μm, each capable of rapidly flipping between two static states with tilt angles of θb−off=−12° (off-state) and θb−on=+12° (on-state) relative to the surface normal on the x′-axis in a few microseconds [Fig. 1(b)]. A “knife-edge” binary pattern, acting as a half-plane filter in the spatial frequency domain, is loaded on the DMD. By maintaining the “off”-state micromirrors and flipping the “on”-state micromirrors, the DMD serves as a coded dynamic 2D blazed grating. Among the many DMD-generated diffraction orders, those that are distributed along and in proximity to the horizontal axis are selected to pass through the stereoscopic objective lens, subsequently landing on spatially separated positions on the IIP. Finally, they are imaged by a relay system, consisting of two lenses (Thorlabs, AC508-100-A and AC508-75-A), to a CMOS camera (Optronis, CP70-1HS-M-1900) that is rotated by approximately 34° to accommodate the recording of a maximal number of diffraction orders by the sensor.

In the following, we assume a unit magnification of the stereoscopic objective lens and the perfect synchronization of the micromirrors’ flipping motion. We also ignore aberrations and the finite numerical aperture of the objective lens and relay system. The image of the transient scene created at the IIP by the front optics and beamsplitter is denoted by O(x,y,t). Under the action of the stereoscopic objective lens, the Fourier plane of the IIP is located at the DMD plane. Subsequently, the modulated field at the DMD plane is expressed by UDMD(x′,y′,t)=O˜(x′λf,y′λf,t)[M(x′,y′,t)⊗A(x′,y′)]K(x′,y′).(1)

Here, λ denotes the wavelength of the continuous-wave laser. f denotes the focal length of the stereoscopic objective lens. “⊗” denotes the 2D convolution. O˜(fx,fy,t) denotes the Fourier transform of the transient scene at the IIP, where the relationships between spatial frequency coordinates and DMD-plane spatial coordinates are expressed as fx=x′λf and fy=y′λf. The three functions M(·), A(·), and K(·) are defined as M(x′,y′,t)=rect(x′−y′2w,x′+y′2w)exp{i2πx′sin[2θb(t)]λ},(2)A(x′,y′)=comb(x′p,y′p)+comb(x′p−12,y′p−12),(3)K(x′,y′)=rect(x′−xk′LKE,y′Ly′).(4)

The function M(·) describes the reflection from a square DMD micromirror (of width w) that is both tilted at a time-varying angle of θb(t) relative to normal about the y′ axis and oriented at 45° with respect to the x′ axis [Fig. 1(b)]. The function A(·) describes the aggregate positioning of the DMD micromirrors as two superimposed rectangular grids of pitch p, where the second grid is shifted by a distance p2 along both the x′ and y′ axes [Fig. 1(b)]. As a result, the full micromirror array is described by the 2D convolution of M(·) and A(·). Finally, the function K(·) describes the finite rectangular aperture of the DMD, with the width Lx′ and the height Ly′ restricted for schlieren imaging to a horizontal extent LKE and centered horizontally on the coordinate xk′=Lx′−LKE2.

After reflection from the DMD, the stereoscopic lens induces another Fourier transform, mapping the DMD plane to the IIP. Subsequently, the complex field at the IIP is expressed by UIIP(x,y,t)=O(−x,−y,t)⊗[M˜(xλf,yλf,t)A˜(xλf,yλf)]⊗K˜(xλf,yλf),(5)where the functions M˜(·), A˜(·), and K˜(·) denote the 2D Fourier transforms of M(·), A(·), and K(·), respectively. The appearance of O(−x,−y,t) in Eq. (5) represents the inverted image of the transient scene under unit magnification induced by the folded 4f imaging system. The commutativity and associativity of “⊗”, together with the properties of the comb(·) function, then allow for the re-expression of Eq. (5), up to proportionality, as $$\begin{gathered} UIIP(x,y,t)∝∑m=−∞∞∑n=−∞∞{1+cos[π(m+n)]} \\ ×sinc{w2p{m−n−pλsin[2θb(t)]}×w2p{m+n−pλsin[2θb(t)]}} \\ ×S(x−λfmp,y−λfnp,t), \end{gathered}$$(6)where the function S(·) is the schlieren image of the transient scene, defined as S(x,y,t)=O(−x,−y,t)⊗[sinc(xLKEλf,yLy′λf)⁢exp(−iπx2xk′λf)].(7)

Equation (6) thus defines the image produced in the IIP as a 2D array of shifted copies of the schlieren image of the transient scene, each indexed according to integer diffraction order coordinates (m,n) and modulated by an intensity envelope with a time-varying position parameterized by the micromirror tilt angle θb(t). Due to the presence of the term {1+cos[π(m+n)]} in Eq. (6), diffraction orders with m+n odd are suppressed, producing an effective diffraction pattern of a rectangular grid with a 45° orientation.

Taking the squared modulus of Eq. (6) produces the time-dependent intensity signal IIIP(x,y,t) imaged by DRUMS photography. Under the assumption that the diffraction orders produced by the DMD are well-spaced relative to the FOV presented by the copies of the schlieren images, the intensity signal can be approximated as a sum of intensities of separate diffraction orders: $$\begin{gathered} IIIP(x,y,t)≈∑m,n∈Zmod(m+n,2)=0|S(x−λfmp,y−λfnp,t)|2 \\ ×sinc2{w2p{m−n−pλsin[2θb(t)]}×w2p{m+n−pλsin[2θb(t)]}}. \end{gathered}$$(8)

Equation (8) thus provides the foundation for DRUMS photography’s image acquisition, with the rapid time-dependent positioning of the sinc2(·) intensity envelope serving as the basis for diffraction-gated ultrahigh-speed imaging. Finally, after the relay of the intensity signal to the camera, image acquisition takes place via the time integration of IIIP(x,y,t).

The sequence depth of DRUMS photography is governed by the spacing of diffraction orders on the IIP, together with the swept range of the sinc2(·) envelope in Eq. (8). Given the limits θb−off≤θb(t)≤θb−on obeyed by the micromirrors, the range of horizontal diffraction orders swept by the peak of the sinc2(·) envelope can be seen to obey pλsin2θb−off≤m≤pλsin2θb−on. In this work, the range of on-axis diffraction orders is expressed by n=0 and m=0,±2,±4,±6, and ±8. Similarly, characterized by Eq. (8), the range of swept off-axis diffraction orders is expressed by n=±1 and m=±1,±3,±5,±7. In practice, however, the six diffraction orders (±8,1), (±7,1), and (±7,−1) were found to be too close to the beam reflection directions in the static “off” and “on” states of the micromirrors, thus resulting in a high background. As a result, these diffraction orders were blocked at the IIP. Noting the equivalence of data produced by off-axis diffraction orders (m,±1), we combined the corresponding images to improve the signal level and mitigate the intensity discrepancy between on-axis and off-axis images. Thus, DRUMS photography achieves a sequence depth of 13 frames.

Assuming that the zero overlap occurs between images produced by adjacent diffraction orders, the FOV of DRUMS photography is determined by the magnification ratio of the front optics Mobj and the area allotted to the image associated with each diffraction order at the IIP. Further assuming that these images may each cover a disk with a diameter equal to the minimum space between diffraction orders, the input-space FOV diameter may be expressed by DFOV=2λfpMobj.(9)

By substituting the value f=45mm and Mobj=2.2–4.0 corresponding to our experiments, the FOV diameter of DRUMS photography was calculated to be DFOV=0.70–1.26mm. It should be noted that a variety of choices of FOV shape and size are compatible with DRUMS photography, depending on how identical regions surrounding each diffraction order are allowed to partition the IIP via the choice of field stop in the optical setup. For example, by partitioning the IIP into horizontally oriented rectangular regions centered on each diffraction order [see the inset in Fig. 1(a)] and using Mobj=4.0 front optics, the DRUMS photography system attained an FOV of size 1.0mm×0.5mm (width × height).

Stemming from the diffraction gating mechanism, the frame rate of DRUMS photography is determined by the time interval separating the arrival of the diffraction envelope’s peak intensity at adjacent diffraction orders along the x axis of the IIP. Owing to the shape of the sinc2(·) envelope, the arrival time tm seen by a diffraction order (m,n) can be observed from Eq. (8) to satisfy m−pλsin[2θb(tm)]=0.(10)

Making use of the small angle approximation, together with the assumption that the speed of mirror flipping dθbdt is constant during the interval [tmmin,tmmax] for the range m=mmin,…,mmax of recorded on-axis diffraction orders, the framerate of DRUMS photography may be expressed as rDRUMS=mmax−mmintmmax−tmmin=2pλdθbdt.(11)

For our characterization of the framerate of DRUMS photography, the values of tmmax and tmmin were measured directly in the time domain using a photodiode that recorded light from the desired diffraction order during mirror flipping. Consequently, the time interval spanning diffraction orders (−6,0) and (6,0) was measured at t6−t−6=1.23μs, yielding a framerate of 9.8 Mfps across 13 frames.

We numerically simulated DRUMS photography using an animation of “Pacman” [Fig. 2(a)] as the ground truth scene. The animation consisted of 13 frames, each with dimensions of 2000 pixel×4000 pixel, which were employed to simulate a 1.0mm×0.5mm FOV. To mimic the configuration of a DMD, an array of 93×93 micromirrors was simulated with each micromirror, modeled as a perfectly reflective 45°-oriented square covering 44 pixel ×44 pixel of area at the Fourier plane. The simulation of micromirror flipping was achieved by specifying a linear phase profile across each mirror. The fill factor of the micromirror array, chosen to mimic that of the DMD used in the DRUMS photography system, was set at 92%^[33]. To illustrate the operation of the simulated DMD array, Fig. 2(b) depicts five close-up phase profiles that correspond to the frames in Fig. 2(a).

For schlieren imaging, a knife-edge filter was applied to the 2D Fourier spectrum of each frame in the ground truth. As an example, Fig. 2(c) illustrates the 2D Fourier spectrum of Frame 7, with the blocked portion of the spectrum represented by the green-dash-filled rectangle. Each filtered spectrum, multiplied with the corresponding DMD modulation profile, was inversely Fourier transformed to produce the schlieren image. Five representative DRUMS photography results are depicted in Fig. 2(d), with the full animated sequence shown in Visualization 1.

To illustrate the effect of schlieren imaging, line profiles extracted from simulation results of Frame 1 and Frame 10 [shown by the light blue horizontal lines in Fig. 2(d)] are compared to the corresponding ground truth profiles in Figs. 2(e) and 2(f), respectively. As the line profiles show, the vertical orientation of the knife-edge, together with the blocking of the central DC component of the 2D Fourier spectrum, produces simulated schlieren images that isolate vertically oriented edge features in the ground truth.

We positioned a negative USAF 1951 resolution target (Thorlabs, R3L1S4N) at the object plane under continuous-wave illumination to assess DRUMS photography’s spatial resolution. A representative non-schlieren image of the target recorded by DRUM photography is shown in Fig. 3(a), with the entire 13-frame sequence recorded by DRUMS photography shown in Fig. 3(b). The system can resolve Group 6 Element 2, corresponding to a spatial resolution of 7 µm. To illustrate the effect of knife-edge filtering by the DMD, line profiles [shown by the light blue line in Fig. 3(b)] extracted from images recorded by DRUM photography and DRUMS photography are presented in Fig. 3(c), corresponding to frames extracted by the (−6,0) diffraction order at time t=0μs. Similar to the results of the numerical simulation, images recorded by DRUMS photography show the isolation of edge features, demonstrating the system’s schlieren imaging capability.

We then conducted proof-of-concept experiments by imaging a fracture defect of a glass microscope slide illuminated by the 473 nm light source modulated with a 2.5 Hz square wave with a 50% duty cycle. A bright-field image recorded by DRUM photography at t=0μs is shown in Fig. 3(d), with selected frames from the sequence recorded by DRUMS photography shown in Fig. 3(e) and shown fully in Visualization 2. The phase variation caused by the surface discontinuities and rough features, while hardly discernable in the bright-field result, is revealed by DRUMS photography. Finally, the integrated intensity of a selected local region [marked by a light blue rectangle shown in Fig. 3(e)] is provided in Fig. 3(f), showing DRUMS photography’s ability to resolve the details of intensity-modulated illumination.

We employed DRUMS photography to investigate the interaction between femtosecond laser pulses and liquid water. In the experimental setup depicted in Fig. 4(a), a single pump pulse (1035 nm wavelength; 350 fs pulse duration; 7 µJ pulse energy) was generated by a femtosecond laser (Huaray, HR-Femto-10) and focused into distilled water contained in a glass cuvette using a lens with a 15 mm focal length (Thorlabs, LB1092). The tightly focused beam ionized the distilled water creating a plasma channel at the focus. Five representative frames recorded by DRUMS photography of the refractive index perturbation caused by the laser-induced breakdown are shown in Fig. 4(b), with the full evolution shown in Visualization 3. The time history of the channel length is illustrated in Fig. 4(c), showing that the channel initiates with a length of 434 µm at 0.23 µs and subsequently decreases to 70 µm at 1.23 µs.

We have developed DRUMS photography, a single-shot 2D ultrahigh-speed schlieren imaging technique capable of a frame rate of 9.8 Mfps and a sequence depth of 13 frames. By leveraging the abilities afforded by a folded 4f imaging system for optical processing together with the inter-pattern transitions of a DMD, this system creates a swept diffraction-based time gate that sequentially extracts frames from 2D diffraction orders. In contrast to traditional DMD-based optical processors^[34–38] that operate via sequences of statically displayed patterns at the Fourier plane, the inter-pattern transitions utilized by DRUMS photography realize a form of intrinsically dynamic Fourier plane filtering that enables the simultaneous implementation of ultrahigh-speed imaging with 4f optical processing. Meanwhile, leveraging the DMD’s programming flexibility, a “knife-edge” filter is implemented to effectively block half of the spatial frequency content along with the DC component to sense the phase gradient of the transient scene. We carried out theoretical derivation, numerical simulation, and experimental validation of DRUMS photography. The technique was applied to the study of laser-liquid interactions.

DRUMS photography represents one of the fastest schlieren photography systems. Unlike existing setups in this category^[39,40], DRUMS photography is developed upon the normal operation of a camera. It retains the advantage of high sensitivity in many scientific-grade cameras (e.g., electron-multiplying CCD cameras and scientific CMOS cameras) and endows them with ultrahigh imaging speeds. In contrast to computational high-speed imaging techniques that employ spatial domain encoding and reconstruction algorithms for the recovery of transient scenes, DRUMS photography avoids the computational overheads associated with compressive reconstruction, as well as the limitations in spatial resolution imposed by the use of spatial domain coded apertures and the inherently ill-posed nature of inverse problems confronted by such techniques in recovering high-dimensionality datasets from low-dimensionality information. Moreover, implemented in a single camera, DRUMS photography embodies a compact and cost-efficient system compared to its counterparts requiring multiple cameras and/or active optical elements^[41,42]. Furthermore, the programmable nature of the deployed DMD—as a coded 2D dynamic blazed grating^[43]—permits the customization of imaging parameters tailored to specific experimental requirements, further enhancing the system’s adaptability. In terms of repeatability and long-term stability, DRUMS photography benefits directly from the well-established reliability and robustness of DMD hardware, with demonstrated operation lifetimes exceeding 100,000 h and 3 trillion display cycles^[44]. Benefitting from a non-mechanical knife-edge filter, DRUMS photography features high accuracy and flexibility in system alignment and calibration. Finally, considering the broad responsive spectrum of the DMD, DRUMS photography possesses high adaptability to dynamic scenes at different wavelengths.

Future work will be carried out in the following three aspects. First, we plan to further enhance the system’s imaging speed using micro-electro-mechanical system technology with a shorter inter-pattern transition^[45]. Second, although demonstrated only for schlieren imaging, DRUMS photography could be applied to phase-sensitive microscopy by loading a rotationally symmetric filter on the DMD^[46]. Finally, we plan to apply DRUMS photography to study the impact of tissue heterogeneity on laser ablation for minimally invasive surgeries^[47].