6G networks are expected to provide wireless communications with extremely high capacity and transmission speed for the Internet of Everything [1]. Traditional radio wireless communications are allocated with limited bandwidth, which will be quickly exhausted with the ever-increasing demand. Optical wireless communication (OWC), especially visible light communication (VLC), is an emerging technique utilizing the unlicensed ultra-high bandwidth within the optical frequency range [2–4] and holds great potential to bring breakthroughs to next-generation high-speed networks. Since VLC can be easily deployed on existing illumination systems such as indoor lightings and car headlights, it has been brought to various applications spanning from civil (e.g., indoor LiFi networks [5] and secure transmission in environments with high electromagnetic interference [6]) to military tasks (e.g., underwater blue–green light communication with over gigabits data rates [7], and inter-satellite long-range laser communication [8]).

It is foreseen that sensing functionalities should serve as a basic service in 6G, enabling future networks to see the physical world and facilitate applications such as the Internet of Vehicles (IoV) and virtual reality (VR). In addition, communication and sensing systems can be mutually beneficial; for example, employing localization to assist beam-steered communication allows the integrated system to fulfill the broad coverage and high-speed requirements. Driven by this demand, efforts have been made towards high-resolution localization in electromagnetic wave wireless communication using radio [9], millimeter [10,11], and terahertz waves [12], achieving localization accuracy from several meters to tens of centimeters. Light communication, which works on much shorter wavelengths, is promising to achieve higher localization resolution. Nevertheless, integrated localization in light communication systems has not yet been well established, hampered by the low sensitivity of available photodetectors, which struggle to discern the low-budget reflective photons from the receiver in the presence of ambient light. Previous visible light positioning techniques are primarily developed in two categories: camera-based and photodetector (PD)-based methods. The former techniques adopt additional array detectors to acquire the sensing information, whereas the sensing and communication functionalities are achieved by two independent systems and are costly in the hardware budget [13,14]. PD-based methods retrieve the position information from received signals by interpreting the arriving time delay or channel attenuation responding to single or multiple separately placed transmitters that encode unique identification signals [15,16]. Yet, these methods are vulnerable to ambient noise and only provide centimeter-level localization resolution. In addition, the multi-transmitter approaches greatly increase the system complexity with the demand for stringent clock synchronization among transmitters. The generalized and efficient implementation of a deeply integrated sensing and transmission system with optical waves remains a challenging yet important task.

In this work, we described a novel solution to visible light integrated sensing and communication by jointly manipulating the spatio-temporal properties of the emitting light field. We employed active spatial modulation on the illumination, in addition to temporal modulation on its intensity, which encodes transmission signals with discrete multi-tone modulation, and achieves high-resolution localization in 3D via computational reconstruction. To improve sensing efficiency, an angular projection imaging scheme supported by compressive sensing [17] was proposed that leads to a 64-fold reduction in measurement time. Following an integrated waveform design, we achieved precise location perception and data transmission with negligible communication performance degradation by processing on the same optical signals, hence greatly augmenting the information throughput. Demonstrating on a free-space optical link covering a wide field of 0.6m×0.6m over 2 m length, we exhibited simultaneous data transmission at ∼3Gbps and 3D localization with <1mm accuracy in horizontal, 4 cm accuracy in vertical at the 39 Hz frame rate. The retrieved localization information was leveraged to achieve beamforming, which adaptively focused and steered the illumination towards the target. We experimentally verified that adaptive beamforming largely enhanced the data rate by 122% at 2 m and 184% at 3 m distance, reaching >6Gbps to enable high-speed wireless VR. Structured illumination with beamforming also improved the localization precision by 2- and 4-fold in lateral and vertical to be millimeter-level in 3D, offering the possibility of more precise and narrower beamforming.

In a typical wireless light communication system, data to be transmitted are encoded into analog signals and drive the illumination light switching at fast speed. Assuming a wide coverage light communication situation, the receiver located within the illumination area collects and measures the streaming light intensity with a photodetector. Since the illumination light is usually designed with a near-uniform or axially symmetric irradiance to maintain similar transmission performance over the working region, the receiver cannot determine its location solely based on the detected light intensity.

To empower spatial sensing ability to PD-based optical communication, leveraging the concept of computational imaging that allows spatial information being demixed from coded single-pixel measurements [18–21], we introduce active spatial encoding to the emitting light to allow spatial-resolving perception. As illustrated in Fig. 1(a), the transmission (Tx) data are temporally encoded on the amplitude (or phase) of the light field as in a standard light communication manner, followed by spatial encoding to produce a sequence of structured illumination (SI). Let u(t) represent the transmission signals and Sk(p) the kth SI pattern, and the spatio-temporal modulated emission light can be expressed as Ek(p,t)=u(t)·Sk(p),k=1,2,…,m. Here, t represents the timestamp, p represents the spatial index, and m is the number of SI patterns. The receiver located at a certain position perceives a small portion of spatially encoded light within its reception field (RF), the integral of which will vary with the position change. Specifically, the received optical signal at the kth SI elapse is $$\begin{gathered} yk(t)∝∫pM(p)·Ek(p,t)dp, \\ M(p)={1,p∈RF0,p∉RF, \end{gathered}$$(1)

Since the Tx signal modulation rate is commonly several orders higher than the SI switch rate, an average smoothing on the received signal leads to the decoupling of the SI-dependent responses from the communication signals: y˜k∝1T∫t∫pM(p)·u(t)·Sk(p)dpdt=u˜∫pM(p)·Sk(p)dp.(2)

Here, u˜ indicates the average intensity of u(t) in each SI pattern elapse T, which is constant for most modulation formats. Given the receiver’s responses to a sequence of SI patterns, the spatial information of the RF can be computationally reconstructed by solving an inverse problem. Meanwhile, the Tx signals at each SI interval can be retrieved by applying a high-pass filter on the received data and conducting the standard demodulation algorithm. Notably, the spatial information is recovered from the same amount of received signals necessitated for data transmission without any increase in sampling budget or hardware complexity. The proposed spatio-temporal modulated optical communication network allows multiple users to directly connect in as working in the broadcast manner, while their position information is kept private from each other and even from the emitter. Since the sensing and communication processes rely on localized spatial modulation and detection, no optical interference or cross-talk occurs across users. These merits make it a promising solution for intelligent and secure interconnection. More importantly, the position sensing functionality will assist the light communication system to perform beamforming, which is essential in practical applications.

We demonstrate the computational localization capability in a free-space VLC system [see Figs. 2(a) and 2(b); see Appendix A]. In addition to temporal encoding to load transmission data, a series of spatial encoding on the illumination is applied to introduce spatial variance. Traditional imaging techniques require dense sampling of the spatial region (e.g., raster scanning or applying a complete set of spatial illuminations). To achieve fast and high-resolution localization in dynamic conditions, we propose an efficient compressive sensing framework to computationally estimate receiver’s position from relatively few coded measurements. Specifically, we design a compressive angular projection imaging scheme, referred to as “CAP,” where stripe patterns along a few directions are used for structured sampling [see Fig. 2(c)]. The concept is similar to the tomographic imaging widely applied in biomedical imaging, which records linear projections of a sample along multiple angles to reduce radiation dose and improve imaging speed [22,23]. Harnessing the geometric symmetry of the receiver’s reception field, the CAP approach merely requires sampling the full field with stripe patterns in two angles to achieve spatial information decoding. The stripe illumination patterns are derived from a complete set of Hadamard basis, which has good tolerance to noise [24,25] and enables projection profiles reconstruction by a simple linear back-projection algorithm. With the sampling pixel resolution of the full field to be N×N, CAP greatly reduces the sampling time to be proportional to 2N, compared to N2 for that of dense sampling. For sampling resolution N=128, CAP requires only a 1.56% sampling ratio (i.e., 512 patterns) and can achieve a frame rate of 39 Hz using a fast spatial light modulator (SLM) working at 20 kHz.

We further explained the procedure of CAP with a representative experimental testcase. The spatially encoded illumination pattern with a pixel resolution of N=128 was projected onto a 0.6m×0.6m area at a distance of 2 m using an SLM and preserved good contrast spanning >0.6m along the depth. The received coded signals by one user [Fig. 2(d)] were divided by the pattern interval and averaged to calculate the location-variant response to each SI pattern [Fig. 2(e)]. The reconstructed projections in the X and Y axes [Fig. 2(f)] were then fused to generate a recovered image as Irec=projX·(projY)T,(3)indicating the reception field of the receiver located within the full field [Fig. 2(g), Appendix C]. The projective intensity profiles can be theoretically modeled by a zero-truncated quadratic function, and a 2D joint fitting method was developed to estimate the center coordinates in X,Y axes and the size of the reception field [see Fig. 2(h)]. With the pre-calibrated knowledge of the full-field size at a certain depth and the receiver’s aperture size, the receiver’s lateral position and depth can be calculated (see Appendix C). The proposed detection algorithm allows for subpixel precision, hence leading to high-resolution 3D localization.

We experimentally characterized the performance of XY localization with CAP by traversing the illumination area at 2 m distance at each of the 8-by-8 subregions [Fig. 3(a)]. The reception field profile at each location is recovered, and the mean squared errors (MSEs) and variations of the estimated X and Y positions calculated on 10 repetitive trials are plotted in Fig. 3(b). The absolute error is mostly below 1 mm throughout the field and no more than 2 mm at the worst case (possibly caused by the system error of manually placing the receiver to the corners). The results indicate that CAP achieves accurate and robust 2D localization. To examine the lateral resolving limit of CAP, we next translated the receiver by small displacements starting at two representative positions highlighted by orange and purple circles in Fig. 3(c). It was observed that the receiver located at two places close to 1 mm could be well distinguished. The experiments validate a high lateral localization resolution of 1/600 of the field size, which is much larger than the spatial modulation resolution.

We further verified the 3D localization performance of CAP by vertically moving the receiver to scan over the depth distance as illustrated in Fig. 3(a). Testing at four distances spanning from 1.6 to 2.2 m, we presented the recovered images indicating the relative position and size of the receiver reception field to the full-field illumination region in Fig. 3(c). The magnified subregions reveal the change in pixel number of the receiver aperture as the depth grows, while the recovered aperture size remains similar for different lateral positions. We examined the depth resolving ability by applying small depth displacements of 2 cm starting at 2 m distance, and the estimation results repeated by 10 trials show that CAP can mostly distinguish positions spaced by 4 cm in depth [see Fig. 3(d)].

After characterizing 3D localization precision of CAP in static context, we next experimented in a practical dynamic scenario where the receiver kept moving within the volume at a fast speed [Fig. 3(f)]. Attributed to the capability of fast and precise 3D localization, CAP achieved real-time tracking of the receiver at 39 fps (frames per second) [see Fig. 3(g)]. The swirling 3D trace within a 10 s duration and representative recovered spatial profiles during the tracing are presented for visualization.

In a light communication system with structured illumination, the received optical signal at each illumination pattern elapse is the product of the communication signals and the response to the SI. Large fluctuations among sensing signals corresponding to different SIs are expected to maintain high signal-to-noise ratio (SNR) for accurate spatial recovery. However, the received signals may be largely attenuated when the sensing responses to certain illumination patterns are low, leading to a decrease in the communication data rate. To address this issue, we developed an integrated waveform design to jointly minimize the illumination power degradation and to preserve high-resolution localization [see Fig. 4(a)]. An intensity rescaling of the SI, i.e., Sk(p)=1−α(1−Sk(p)), is applied by spatial dithering on the SLM to increase the optical intensity collected by the receiver. At the receiver end, a direct current (DC) blocking circuit is utilized to remove the intensity offset of received signals at each illumination pattern elapse. To balance the SNRs of communication signals and sensing signals, an intensity scaling factor of α=1/4 was empirically chosen in all experiments. The cutoff frequency of the DC-block was properly set to be several orders larger than the illumination switching rate to suppress the SI-variant DC components. As a result, the dynamic range of the data acquisition is fully exploited to enable high SNR data transmission, while the SI-relevant intensity variations still guarantee robust spatial reconstruction.

We evaluated the impact of structured illumination on the communication performance in the 2 m free-space VLC link. Discrete multi-tone (DMT) modulated signals with bit and power loading were transmitted, and the received signal intensity and estimated data rates following standard demodulation with respect to different SIs were compared, as shown in Fig. 4(b). We observed small fluctuations in the data rate ranging from 3.16 to 3.52 Gbps when the normalized signal intensity varies from 0.75 to 1. The averaged data rate over different SIs was 3.35 Gbps, only a 4.3% decrease compared with the baseline case without structured illumination [denoted in the dashed line in Fig. 3(b)]. Although the illumination patterns within the receiver’s reception field are largely distinct under different SIs (for example, see the received light field when projecting patterns #5 and #29), we observed similar transmission data rates and tendencies of SNR distribution and bit-power allocations. To conclude, the proposed integrated waveform design ensures that the incorporation of structured illumination into light communication systems will scarcely affect its data transmission performance.

We further characterized the data rates at different spatial positions in 3D. Testing at two distances of 2 and 1.8 m, the measured data rate maps are shown in Fig. 4(c). We saw an increase in the data rate as the distance was reduced because the illumination density gets larger at closer distances. Variations in the data rate across the lateral position are due to the uneven irradiation of the light source and can be avoided by using more uniform illumination. We also investigated the relationship between the bandwidth and data rate to determine the system bandwidth, and the maximum data rates for the position highlighted in Fig. 4(c) at two depths are 3.40 and 3.63 Gbps with 1.0 Gbaud.

Lastly, we demonstrated the performance of data transmission in dynamic conditions. The receiver is swirling within the volume of 0.6m×0.6m×0.6m. Notably, using CAP with integrated optimized spatio-temporal encoded illumination, we achieved simultaneous data transmission and localization. The reconstructed 3D trace of the moving receiver with estimated transmission data rates encoded by color is presented in Fig. 4(d). We see fast and stable data transmission over the illumination volume with data rates modestly varying between 3.2 and 3.8 Gbps as the spatial position changes.

The integration of communication and localization brings the benefit of increased information throughput. Moreover, the perceived knowledge of location can be exploited to promote data transfer. Here, we demonstrated adaptive beamforming at the transmitter that converges the illumination light into a narrower beam and steers towards the target receiver [see Fig. 5(a)]. The position information reconstructed by the receiver could be passed to the transmitter via a low-speed upload link. Optical converging and steering were realized by vertically and laterally translating the collimating lens after the light source, electrically driven by a motorized stage. Since the illumination light density sensed by the receiver is greatly increased, the communication performance will gain from the increase of received SNR (assuming that the PD is not saturated).

We examined the improvement in data rate with adaptive beamforming at four example positions where the irradiance is relatively weak. Given the position information estimated with CAP, the emission light beam was converged to be 1/8 of its original divergence and steered towards the target, and the focused region-of-interest (ROI) covers a 75mm×75mm area around the receiver at 2 m distance. As compared in Fig. 5(b), we observed a 122% maximum increase in the data rate after beamforming, and the maximum data rate acquired exceeded 6 Gbps. For the first position, we also compared the marked improvement in averaged SNR from 12.30 to 18.79 dB and the highest bit allocation number from 4 to 7 bit/(s Hz), and the constellations of 128-QAM and 64-QAM appeared after beamforming, revealing that beamforming allows higher-order modulated signal transmission [see Fig. 5(c)]. The data rate improvement is more remarkable at a longer distance where the full-field illumination is weaker. By projecting structured illumination at 3 m distance, which covers an area of 0.9m×0.9m, the measured data rate at the peripheral of the field merely reached 2.09 Gbps. After adaptive beamforming, the transmission speed was increased by 184% to achieve 5.93 Gbps, which is quite close to that achieved at the shorter distance. So far, the bottleneck of the data rate is the bandwidth of the emitter [26] instead of the optical SNR.

The enhancement of integrated data transmission and localization will play an essential role in high-throughput applications such as VR. The pursuit of cable-free VR devices that cut the wire to the content-generating server requires a network to deliver enormous data at low latency. Here, using our 2 m VLC link for data transmission, we tested two VR videos with high spatio-temporal resolution. The first one is a panoramic 3D video in an equi-rectangular projection of 7680×3840pixels, 120 Hz frame rate, and true color (24-bit), and it is encoded in HEVC format at an averaged compression ratio of 94%. The required data rate for errorless transmission with 7% forward-error-check overhead is computed as 7680×3840×120×24×(1−94%)×(1+7%)=5.45Gbps. The second video is a monoscopic 360° video in an equi-angular cube map of 7200×3840pixels, 120 Hz frame rate, and 24-bit color, requiring a data rate of 5.11 Gbps. Under the wide-field illumination condition, the data transmission becomes erroneous, and the packet loss rates for the two testing videos are ∼38.5% and 34.4%. For visual representation, when the screen content changes or the user’s view rapidly switches [Figs. 5(d) and 5(g)], the display via wide-field VLC transmission exhibits obvious trailing shadows and pixel contamination [Figs. 5(e) and 5(h)]. This can lead to a desynchronized perception of motion in the visual cortex to induce dizziness. The erroneous transmission could even cause content loss [e.g., people become unrecognizable within the cropped ROIs in Figs. 5(e1) and 5(e4)] and fuzziness of fine patterns [Figs. 5(h2) and 5(h3)], whereas, with position sensing-assisted adaptive beamforming, the data rate is enhanced to support lossless transmission of the high-throughput video, offering the possibility for wireless immersive VR experiences [Figs. 5(f) and 5(i)].

Location estimation can also be refined by reprojecting structured illumination within the concentrated ROI. As shown in Fig. 6(a), we generate ROI-focused SI with finer spatial modulation in a subregion surrounding the receiver and consequently achieve more precise spatial resolving. In the refined localization phase, we used a complete set of 2D Hadamard patterns of N=16 to recover the subregion, which takes identical measurements to the two-angle CAP sampling of N=128. Comparing the recovered images, we saw that the reception field was more sharply reconstructed and looked closer to its real shape. We quantitatively evaluated the localization improvement by fixing the receiver and repeatedly measuring its X,Y and depth position. As shown in Fig. 6(b), the estimated values at the testing point after beamforming show a >2-fold reduction in estimation variation. While we bi-directionally translated the receiver in lateral space by a small step of 0.5 mm, the estimated lateral displacements for 10 trials were −0.491±0.006mm and 0.494±0.008mm for the X axis, and −0.482±0.003mm and 0.473±0.004mm for the Y axis [Fig. 6(b)], demonstrating that the refined recovery enables sub-millimeter spatial resolution. The localization improvement is more significant for the Z axis. As shown in Fig. 6(c), while moving the receiver back-and-forth by 5 mm, the estimated Z displacements were −4.01±3.69mm and 4.55±5.48mm, respectively. Two depths of 10 mm distance were well distinguished after refined localization, showing a vertical resolution improvement of ∼4-fold.

Wireless light communication is expected to play an important role in 6G, and it is desired to achieve intelligent sensing functionalities via the same communication channel, yet this pursuit has remained very challenging for visible light. Here, we developed a spatio-temporal modulated structured illumination system with a jointly optimized waveform design and compressive sampling to achieve fast and accurate localization and high-speed data transfer in one integrated system. We demonstrated that the integrated sensing and communication design was mutually beneficial, i.e., adaptive beamforming proceeded assisted by position sensing and gave rise to up to 184% data rate enhancement and approximately 2- to 4-fold localization resolution improvement. The proposed visible light integrated sensing and communication system shows great potential in future wireless VR and other high-speed applications.

In our localization experiments, the effective illumination field, localization resolution, and number of sensing patterns are mutually restricted. The spatial resolution of about 1/600 of the field size with 256 spatial encoded illuminations was demonstrated in the article. For accurate localization in wider field of view, larger quantity of illumination patterns will be required. Higher localization accuracy can be achieved if using more patterned illumination with a finer structure, or faster sensing using fewer illumination patterns with the sacrifice of spatial resolution. To break the limit of spatio-temporal bandwidth, advanced neural network algorithms [27–29] will be developed for illumination pattern design and localization estimation under highly compressed conditions. The CAP method relies on orthogonal stripe projections to estimate the receiver position. This is efficient when the receiver exhibits symmetric geometry. For more complex or irregularly shaped receivers, the reconstructed position tends to represent the centroid of the illuminated region. Including additional projection angles can improve robustness to misalignments and receiver irregularity.

The proposed computational sensing scheme is well applicable to various optical communication systems regardless of signal formats and in diverse contexts such as indoor, inter-vehicle, and underwater scenarios. The implementation is easy to produce, i.e., we only need to add a spatial modulation module to the emitter and the newly coming receiver can be directly connected into the network. Moreover, miniaturized design of the structured illumination module (e.g., micro-DMD chips) and even customized integrated structured light engines will be explored to push this method forward into practical use. We expect that our method would provide an alternative route for integrated sensing and communication of next-generation optical wireless networks.