In the rapidly evolving landscape of human-machine interaction, sensors play a crucial role in bridging the gap between physical and digital realms^[1]. Hands, as the primary organs for tactile interaction with the world, are central to this interface^{[2, 3]}. Capturing the intricate movements and force interactions of hands, particularly fingers, has become increasingly important for creating immersive and responsive systems^[4]. This is especially true in the era of virtual reality (VR) and augmented reality (AR)^{[5, 6]}, where the accurate sensing and interpretation of finger movements and force interactions are fundamental to developing robust applications across various fields, including telemedicine^[7], smart industrial processes^[8], and immersive entertainment^[9]. The index finger, owing to its unique physiological structure with flexibility and independent muscle attachment, has always been a key digit in human mechanical interactions^{[10, 11]}. Its importance has been further amplified in current touch-screen dominated era, where it serves as the primary input method for a wide array of devices and applications^[12]. Despite its crucial role, accurately recognizing the index finger’s joint bending state and force interactions poses considerable challenges^[13]. These difficulties stem from the complex biomechanics of the finger and the need for highly integrated, lightweight sensors capable of capturing its nuanced movements and forces.

Sensing of fingers primarily encompasses three phases: self-bending perception, contact-based pressure sensing, which can be further divided into normal pressure and sweeping sensing^[14]. Researchers have strides in both bending and pressure sensing individually^[15−18]. To fully capitalize on the potential of finger-based interactions, it is essential to integrate various sensing functionalities into a unified system capable of capturing both bending states and pressure information. However, this integration also faces challenges. The primary issue stems from the need to locate all sensing units on the palmar side of the hand to effectively acquire contact force information. Traditionally, bending state sensors are placed on the dorsal side of finger joints to better utilize strain^[19]. This placement makes integration with pressure sensors challenging due to undesired stress and strain introduced at cross-surface connections during finger movement. Consequently, there is a pressing need to develop inward bending sensors that can be integrated with pressure sensors on the same side, enabling multi-scenario application recognition based on three-phase identification^[20]. Typically, inward bending detection can be realized through the construction of asymmetrical structures. For instance, Li and colleagues created directional bending sensors by simply interlocking two fibers to form a knot on a flexible printed circuit board, enabling vector sensing within a −90° to +90° range^[21]. While the built-in structures also work, Bai and team developed a Janus structure using MXene within an elastic body, which facilitated the detection and differentiation of both inward and outward bending forces^[22]. Therefore, these highly sensitive directional bending sensors offer simplification and opportunities for the development of palmar sensing arrays.

Another issue in creating integrated finger sensing patch is the acquisition of two types of contact-based tactile information. Traditional flexible substrate-based sensors often introduce interference due to deformation of the underlying material, compromising measurement accuracy^{[23, 24]}. Magnetoelectric generators (MEG) represent a self-powered sensing technology based on the recently discovered giant magnetoelastic effect in soft systems^[25]. Comprising a magnetoelastic body and signal transmission coils, MEG is well-suited for the fabrication of out-of-plane contact sensors, characterized by short response times and high signal-to-noise ratios (SNR). Chen and colleagues were the first to demonstrate the application of the giant magnetoelastic effect in flexible sensors, achieving the detection of human pulse waves through the use of a magnetoelastic thin film^[26]. Subsequently, Ding et al. constructed a sensing array sensitive to sliding motion by utilizing magnetic microcolumns with in-plane magnetization direction^[27]. Due to the distinct sensing and force interaction mechanisms, MEG-based sensors exhibit tremendous potential in the integration of multimodal sensing.

Herein, we present a facile approach to address these challenges by developing a highly integrated, flexible electronic skin designed as an intelligent patch for the index finger. Our device incorporates four sensors based on resistive and giant magnetoelastic effects, enabling simultaneous collection of surface pressure and finger joint bending information. This innovative design allows for the identification of three phases of finger action: the finger’s bending state, dynamic contact behavior, and gripping force. By integrating resistive carbon nanotubes (CNT)/polydimethylsiloxane (PDMS) films for bending and dynamic contact sensing with self-powered magnetoelastic sensors for pressure detection, our system achieves comprehensive tactile perception with high sensitivity and rapid response. This integrated approach not only overcomes the limitations of traditional sensing methods but also paves the way for more nuanced and accurate finger-based interactions in VR/AR environments, medical diagnostics, and industrial applications.

For the microcilia-based bending sensor, carbonyl iron particles (CIP) were obtained from Sigma Aldrich (USA). Polydimethylsiloxane (PDMS) base and curing agent (Sylgard 184 kit) were sourced from Dow Corning, USA, and were uniformly mixed with CIP to prepare the magnetic composite for microcilia formation. This mixture was spin-coated onto a 5 × 5 cm² glass slide to create an uncured thin film. The film was then placed on a NdFeB permanent magnet (5 × 5 × 2.5 cm³) with a surface magnetic field of 400 mT, procured from China Jiuji Co., Ltd. Under the combined effects of gravity, surface tension, and magnetic force, the mixture self-assembled into a microcilia structure guided by the magnetic field. The magnet and the PDMS/CIP composite were then transferred to an oven set at 40 °C for 2 h to achieve complete curing. Subsequently, a conductive surface network was created by spray-coating the structure with conductive carbon black (VXC-72R) dispersed in cyclohexane. The carbon black, with an average particle diameter of 30 nm, was provided by Cabot Co., USA, while the cyclohexane was sourced from Damao Chemical Reagent Factory (Tianjin, China). This spray-coating process aimed to establish a conductive network on the surface of the microcilia structure, realizing its strain-sensing capabilities. (Fig. 1(a)).

For the micropillar array-based MEG sensor, a plastic mold with pre-designed micro-hole arrays was utilized as a template for micropillar (NdFeB/Ecoflex) preparation. Neodymium-iron-boron (NdFeB) particles were sourced from Magnequench, China, while EcoFlex was obtained from Smooth-On, Inc., USA. The EcoFlex gel and NdFeB particles were uniformly mixed at a specific mass ratio. This composite was then poured into the plastic mold and cured on a hot plate at 80 °C for 20 min. After solidification, the cured NdFeB/Ecoflex composite was removed from the mold and placed in a magnetic field with a strength of approximately 3 T and desired orientations for magnetization (Fig. 1(b)).

The copper coil was laser patterned using a laser engraving machine (LPKF ProtoLaser U4, LPKF Laser & Electronics AG, Germany). Each loop had a width of approximately 70 µm, with a distance of about 80 µm between adjacent conductive lines. The coil consisted of 30 turns, separated by an insulating polyimide film. Post-engraving, the coil was treated in citric acid to remove the oxidized layer, and conductive silver glue was applied to unite both layers. Finally, plastic encapsulation was used to protect the coil from oxidation during application.

Upon completion of the individual components, a finger-shaped substrate was carved from a plastic template. The depth of this substrate was set at approximately 200 μm, a dimension carefully chosen to balance wearable comfort with the need to accommodate the embedded coil thickness. Double-sided adhesive tape was used to attach the device to the index finger. PDMS, mixed at a 10 : 1 ratio, was applied via a scraping method to adhere the bending sensor, MEG sensor, and copper coils to the fingertip, DIP joint, MIP joint, and the inside of the proximal phalange of the index finger, respectively (Fig. 1(c)).

The SEM images were obtained using field-emission scanning electron microscopy (FE-SEM, Carl Zeiss, Germany). Variations in magnetic flux and corresponding field intensity were measured using a commercial Gauss meter (Model HT20, HengTong Magnetoelectricity Ltd., Shanghai, China). For testing the bending sensor, real-time electrical signal measurements were conducted by covering both ends of the device with silver paste and adhesive copper foil to ensure good electrical connections. These connections were secured with tape and linked to an external digital multimeter (34460 A, Keysight) to record resistance changes. The device was vertically fixed to a mechanical test system (Mark-10, China) without applying external strain. As the unfixed end moved, the PET substrate bent along with the device, and the corresponding resistance changes were recorded in real time. By defining the bending direction, both inward and outward bending could be achieved, and the respective resistance changes were monitored. To assess cyclic stretching performance, the stretching machine conducted signal testing under −120° to 120° bending loads. Additionally, 400 stretching cycles were performed under 8 different inward bending conditions to confirm the long-term stability of the sensor. The response time and responsiveness of the prepared device were determined by controlling the periods of cyclic stretching and release.

For the MEG sensor test, the assembly was fixed on a mechanical testing system (Mark-10, China). The mechanical indenter was set to deflect the apex of the micropillars horizontally at various loading speeds. Simultaneously, the electrical signals generated in the coil were recorded by an external voltmeter (34460 A, Keysight). Further tests were conducted to evaluate the MEG sensor’s response to different materials. This was achieved by attaching 1 cm thick samples of various materials (iron block, glass, PDMS, Ecoflex) to the indenter. Cyclic pressure loading was then applied at a speed of 500 mm∙min⁻¹ to obtain the MEG sensor's pressure response for these different materials. To assess the device's sensing performance for various surface roughnesses, it was fixed to a platform. A motorized platform, controlled by a system (MAR 100–90, Zolix Instruments Co. Ltd., China), was used to sweep the sensor across sandpaper and printing paper attached to glass slides. The voltage signals were recorded using the external voltmeter. Additionally, practical tests were conducted using four digital multimeters to capture natural gripping signals from a human right index finger wearing the device.

Moreover, the MEG sensor demonstrates a specific response to sliding motions due to the tilting of magnetized micropillars caused by frictional forces, resulting in changes in magnetic flux. Notably, the sensor can distinguish between smooth and rough surfaces during sliding. This differentiation stems from the vertical oscillations of the micropillars as they traverse surface irregularities of rough textures, manifesting as fluctuations in the signal output. Moreover, this force interaction-based sensing behavior responds to different materials. When the indenter is replaced with 1 cm thick blocks of iron, glass, PDMS (10 : 1 ratio), and ecoflex, and subjected to the same speed (500 mm∙min⁻¹) and maximum displacement (1.8 mm) compression, distinctly different electrical responses are observed. Notably, the signals corresponding to the rebound process increase with the rising Young’s modulus of the materials, attributable to the varying energy storage and release characteristics of materials with different elastic properties during the compression and rebound cycle. The ability to differentiate between materials and surface textures through force interactions demonstrates the sensor’s potential for advanced tactile perception. This capability could be valuable in applications such as robotic manipulation, virtual reality interfaces, quality control in manufacturing, and medical diagnostics. By providing rich, multi-dimensional tactile information, including force magnitude, frequency, material properties, and surface textures, this MEG sensor opens up possibilities for creating more sophisticated and responsive human-machine interfaces across various fields.

The human hand plays a crucial role in mechanical human-machine interactions, serving as a key approach for realizing virtual reality applications across various domains, including entertainment^[28], medical diagnostics^[29], and social interactions^[30] (Fig. 2(a)). Among the digits, the index finger bears a significant functional responsibility due to its unique physiological structure, featuring independent muscle attachment and high dexterity^[31]. Its actions can be categorized into three distinct phases: contact-based pressure, contact-based sweeping, and self-bending as shown in Fig. 2(b). The recognition of these three phases is essential for creating intuitive and responsive human-machine interfaces. Contact-based normal pressure provides information about object manipulation and force control, crucial for tasks ranging from delicate medical procedures to industrial assembly. Contact-based sweeping enables direct interaction with digital interfaces, enhancing the user experience in virtual and augmented reality environments. Self-bending recognition allows for gesture-based controls and assessment, valuable in applications such as rehabilitation and gesture-based user interfaces. To bridge this multi-dimensional need, we have developed a smart finger patch (SFP) with coupled magnetoelastic and resistive flexible sensors. This device comprises four sensing units integrated on a PDMS substrate, positioned to correspond with the finger pulp, the distal interphalangeal (DIP) joint, and the middle interphalangeal (MIP) joint, and the muscles on the inside of the proximal phalange of the index finger. As illustrated in Fig. 2(c), the device incorporates magnetized micropillar arrays/flexible coils and microcilia-based inward bending sensors, which correspond to dynamic phases 1 and 2, and state phase 3, respectively.

The magnetized micropillar arrays and flexible coils operate based on the giant magnetoelastic effect, a phenomenon that allows for significant changes in the magnetic field within an elastic system in response to applied stress. This effect arises from the strong coupling between the mechanical and magnetic properties of certain materials. The magnetic elastomer is fabricated from NdFeB-doped ecoflex, molded into a 3 × 3 array of micropillars with a diameter of 1 mm and height of 2 mm, as shown in the scanning electron microscope image in Fig. 2(d). The giant magnetoelastic effect enables the conversion of mechanical stress into localized magnetic field changes, which can be further combined with magnetic induction for power generation. When subjected to external forces, the deformation of the magnetized elastomer induces changes in the magnetic field distribution, which in turn generates current in the underlying coils through electromagnetic induction, providing real-time feedback on the forces acting on the micropillar array. Due to the out-of-plane magnetization direction, when a pillar undergoes typical lateral deflection, it produces a positive signal. Upon force release, as the micropillar returns to its initial position, an opposite signal is generated. Consequently, the magnetized micropillars exhibit a dynamic response to interactive pressure. On the other hand, the inward bending-responsive resistive sensors are based on densely packed, surface-conductive microcilia. During inward bending, these microcilia undergo mutual compression and increased contact, resulting in additional conductive pathways. This leads to a decrease in electrical resistance. The two resistive sensors offer accurate measurements of finger joint angles, enabling detailed tracking of finger posture and movement. Therefore, based on the two types of sensing units, the SFP can achieve perception of both self-body movements and external interaction forces on the index finger. This dual-sensing capability enables comprehensive monitoring of finger actions and interactions, providing a robust platform for advanced human-machine interfaces.

The microcilia-based bending sensors offer precise detection of finger joint angles, allowing for accurate tracking of self-initiated movements and gestures. These sensors can distinguish between various degrees of bending, providing detailed information about the finger’s posture and movement patterns. As described in the "Materials & preparation" Section, the main structure for achieving inward bending is realized through the surface-conductive microcilia structure of the sensor. The microcilia structure is obtained through self-assembly of uncured PDMS/CIP mixture in a magnetic field. After curing, a conductive network is further constructed by spraying conductive carbon black nanoparticles dispersed in cyclohexane solution. Cyclohexane has a swelling effect on PDMS, therefore, during the spraying process, combined with the impact force of the spray, the carbon nanoparticles achieve more secure anchoring. This allows for easy changes in the conductive network through contact rather than just compression, when inward bending occurs, resulting in a decrease in resistance. Conversely, during outward bending, the separation of surface microstructures and their own tensile deformation lead to an increase in resistance. Consequently, the device exhibits significant relative resistance changes as electrical signal responses in the range of −120° (outward bending) to 120° (inward bending), as shown in Fig. 3(a). At −120°, there is a 148.2% relative positive resistance change, while at 120°, there is a 73.5% resistance decrease. Given the physiological constraints of human joint movement, which limit bending to a single direction, the following discussion focuses on inward bending. As shown in the inset of Fig. 3(b), when one end is fixed and the other is free, the angle between the free end and the initial position is defined as the angle change during the inward bending process. It can be seen that the reported bending sensor has high sensitivity, showing distinguishable resistance change rates of 7%, 18%, 35%, 42%, 47%, 54%, 59%, and 73% when loaded with 5°, 10°, 20°, 40°, 60°, 80°, 90°, and 120° inward bending, respectively. To ensure the stability of the bending sensor performance, we conducted 50 measurements at each bending angle and found that the signals are consistent, ensuring robustness in practical testing applications. Then, the response time and recovery time were measured (Fig. 3(c)). The response time and recovery time are 120 and 110 ms, respectively, enabling real-time, high-frequency detection with fast response capability. To ensure that the sensor’s detection range and accuracy cover the requirements for index finger bending states, actual wearing tests were conducted (Fig. 3(d)). To ensure firm conformal contact between the sensor and human skin, the prepared bending sensor is attached to a thin PU adhesive. Our sensors are then attached to the two joints of the index finger, accurately recording the motion bending state during the gripping process. Under four states of fully open, relaxed, attempted grip, and tight grip, state changes can be accurately and stably recorded.

As the primary contributors to dynamic sensing, the MEG sensors constructed with magnetized micropillar arrays are inspired by the sensing principles of human cilia. These sensors enable high-sensitivity detection of external forces and pressures applied to the finger. This capability allows for the quantification of interaction forces during object manipulation and tactile exploration of surfaces with varying textures and materials. The structure of these micropillars is obtained in a single step through a simple templating method, followed by magnetization to achieve directional magnetic fields (see "Materials & preparation" Section for detailed fabrication process, see optimization process in supplementary information text, Figs. S1−S2). The NdFeB/Ecoflex micropillars have a relatively low Young’s modulus, making them more susceptible to deformation under external forces. To optimize gripping functionality and force sensing, we designed micropillars with a 1 mm diameter and 2 mm height, featuring a dome-shaped top. This geometry accommodates various force directions encountered during diverse gripping processes. Recognizing that finger-object interactions involve distributed forces rather than point contacts, we implemented a 3 × 3 array of these micropillars. This configuration enables force sensing over a larger area, providing comprehensive grip force representation. The dome structure enhances force distribution and multi-directional sensing, while the array design improves overall sensitivity, allowing for detection of force gradients and patterns across the fingertip.

Fig. 4(a) illustrates the deformation process of the magnetized micropillars under the action of an indenter. Through a magnetic flux tester, it is clearly observable that during the bending process, the spatial distribution of the magnetic field changes, leading to alterations in magnetic flux (Fig. 4(b)). For a single micropillar, it can be expressed by the equation Φ = ∫B·dS, where Φ is the magnetic flux, B is the local magnetic field strength, and S is the effective area of the coil. As the magnetic flux within the coil region changes, an electromotive force (EMF, ε) is generated according to Faraday’s law of induction ε = −N∙(dΦ/dt), where N is the number of turns in the coil, and dΦ/dt is the rate of change of magnetic flux. The micropillars are magnetized with the north pole at the top and the south pole at the bottom. When a downward force is applied to the micropillars, the north pole moves closer to the lower coil, resulting in a negative value for ΔΦ_B. Consequently, an immediate positive voltage peak appears in the induced ε within the coil. Due to the excellent flexibility and elasticity of the micropillars, a corresponding negative voltage peak is subsequently generated during the recovery period. As shown in Fig. S3, the response time and recovery time of the MEG sensor are approximately 130 ms. These values are comparable to those of the inward bending sensor shown in Fig. 4(c) (120 ms for response and 110 ms for recovery). This similarity in response times between the two sensor types ensures their capability for collaborative and real-time sensing. For the micropillar array in this study, the theoretical induced electromotive force is given by ε = −∑dΦ_Bi/dt, where dΦ_Bi represents the magnetic flux change through the i-th loop. According to the controlling equation, the magnitude of the induced electromotive force is primarily influenced by the frequency of external mechanical stimuli (Δt) and the maximum deformation that determines the magnetic flux change (ΔΦ_B)^[32]. The investigation of the MEG response in the coil at different frequencies reveals that the intensity of the signal increases with higher compression frequencies (Fig. 3(c)), as observed by varying the downward speed of the universal testing machine's indenter. The speed variations tested were 100, 200, and 500 mm∙min⁻¹ for the downward motion, while the upward speed was maintained at 500 mm∙min⁻¹ for all tests. Consequently, the signal intensity during the pillar deformation recovery phase remains similar across all tests. Throughout this process, the maximum displacement of the indenter was set at 1 mm. This consistent recovery speed and displacement limit ensure comparability across different compression frequencies, highlighting the sensor's ability to distinguish varying rates of applied force. Additionally, differences in the electrical signal can be found by altering the deformation magnitude, which changes ΔΦ_B (Fig. 4(d)). As the downward displacement increases from 0.25 to 1 mm, a significant enhancement (approximately 4-fold) in the voltage signal is observed. This enhancement can be attributed to two factors. The larger amplitude of pillar tilting contributes to a more substantial change in the magnetic field. Simultaneously, the compression of the elastomer under the applied force further amplifies this effect. The combination of these factors results in a more pronounced change in the magnetic field, leading to the observed increase in signal strength. Thus, its ability to differentiate various frequencies of force application, as well as magnitudes of deformation, underscores its versatility in capturing complex mechanical interactions. This dual sensitivity to rapid and gradual force changes, coupled with the capability to discern different deformation levels, positions the sensor as an ideal tool for accurately representing the intricate dynamics of finger-object interactions encountered in real-world applications. Such characteristics enable the sensor to provide tactile feedback, closely mimicking the sophisticated sensing capabilities of human fingers.

The MEG sensor also demonstrates a specific response to sliding motions, enabling the detection of surface textures. During sliding, the magnetized micropillars tilt due to frictional forces, causing changes in magnetic flux. This mechanism allows the sensor to distinguish between smooth and rough surfaces, as illustrated in Fig. 4(e). When sliding over rough textures, the micropillars experience vertical oscillations corresponding to surface irregularities, manifesting as fluctuations in the signal output. This capability enhances the sensor's tactile perception, allowing it to discern subtle differences in surface characteristics. Furthermore, the sensor exhibits sensitivity to material hardness. When compressing materials of varying stiffness (iron, glass, PDMS (10 : 1 ratio), and ecoflex) with the same speed (500 mm∙min⁻¹) and a fixed displacement of 1.8 mm, distinctly different electrical responses are observed. Notably, the signals’ intensity increases progressively from the softest to the hardest indenter materials. This phenomenon can be primarily attributed to the relationship between material hardness and magnetic flux changes. The magnetic flux variations are dependent on the alterations in magnetization intensity caused by the elastomer’s deformation. Different indenter hardnesses modify the local stress and strain distributions within the elastomer, thereby affecting the changes in magnetization intensity and, consequently, the magnetic flux variations. Harder materials tend to concentrate stress more locally, leading to more pronounced changes in the magnetic properties of the affected areas. This behavior is consistent with the observed similarities in signal outputs for materials with comparable hardness values, such as glass, metal (iron), and PDMS (10 : 1 ratio). These materials, despite their different compositions, have relatively high and similar hardness values in the context of our sensing mechanism. Consequently, they produce comparable levels of deformation in the magnetized micropillars under the same applied force, resulting in similar signal outputs. In contrast, softer materials like ecoflex generate distinctly different signals due to their significantly lower hardness, further demonstrating the device’s capability to distinguish materials based on substantial differences in hardness values. Interestingly, this effect is more pronounced in the positive peaks observed during the recovery process. This enhanced distinction during recovery can be attributed to the elastic energy stored in the elastomer during compression. Harder materials, which induce more localized and intense deformations, result in greater elastic energy storage. During the recovery phase, this stored energy is released more rapidly and intensely with harder materials, leading to a more significant and rapid change in the magnetic field. This rapid change manifests as higher positive peaks in the sensor's output, providing a indicator of material hardness.

The integration of these two sensing modalities in a single, flexible smart finger patch (SFP) provides a comprehensive solution for sensing of index finger. The SFP is constructed on a PDMS substrate, with strategically placed sensors on the finger pulp, the DIP joint, the MIP joint, and the muscles on the inside of the proximal phalange of the index finger (see "Materials & preparation" Section for details). This design enables the detection of both dynamic force interactions and bending degrees, offering integrated approaches to tactile sensing that overcomes the limitations of visual methods in perceiving mechanical properties of objects. Although visual sensing methods is effective for many tasks, they often fall short in accurately detecting and differentiating subtle mechanical properties such as softness, elasticity, or texture. These properties are crucial in various applications where the physical interaction between a robotic system and its environment is essential. The SFP’s ability to capture both force interactions and bending degrees provides a rich set of data that can complement or even surpass visual information in scenarios where mechanical properties play a key role in object identification and manipulation. Furthermore, both the MEG sensor and the microcilia-based inward bending sensor have been demonstrated to possess high environmental stability. For the MEG sensor, previous studies have shown that PDMS/NdFeB thin-film sensors maintain performance even after prolonged exposure to humid conditions, such as a week of immersion in artificial sweat, and exhibit consistent output across various temperatures^{[26, 27]}. As for the inward bending sensor, the carbon black nanoparticle-modified microcilia array creates a superhydrophobic surface due to the combination of nanoscale carbon black particles with microscale cilia structures. This results in a surface with low surface energy and high roughness, ensuring excellent stability even in aqueous environments^{[33, 34]}. These characteristics collectively contribute to the SFP’s robust performance across a wide range of environmental conditions, making it suitable for diverse applications from everyday wear to more challenging scenarios.

Building upon the SFP’s capabilities, this study explores its application in food quality assessment, particularly in bread recognition. The texture and freshness of bread products are crucial factors in determining their quality and consumer acceptance. Conventional inspection methods are often limited to assessing size and color, but human perception places greater emphasis on "freshness" and texture. The SFP’s high sensitivity and low response time, combined with its dual-sensing modalities, make it admirable for this application. Fig. 5(a) presents characteristic duration (5 s) signals from the four sensors (M1, R1, R2, M3) during the grasping-keeping process for each bread type. The degree of hand bending, which correlates with bread size, can be evaluated through the decrease in signals from R1 and R2. Meanwhile, M1 and M2 provide feedback on the bread's softness or hardness. Notably, softer breads exhibit a distinctive signal pattern: After the initial deformation of the micropillars caused by grasping, there is a partial rebound even while maintaining a fixed grip, manifested as a reverse peak in the signal. In contrast, harder breads do not show this reverse peak before active release, as the micropillars remain in their deformed position. This unique combination of size and texture information allows for specific signal patterns for each bread type: donut, small soft bread, large soft bread, small hard bread, and large hard bread. The temporal data provided by the SFP enables effective classification and recognition of different bread types using even simple machine learning models. To demonstrate this, a straightforward transformer-based model was implemented, utilizing only the encoder part to process the time-series data, followed by fully connected layers for classification, as illustrated in Fig. 5(b). This approach showcases how the high-quality data from our sensor can be easily interpreted to discern subtle variations in signals corresponding to distinct characteristics of different bread types. Thus, such a system can facilitate robot-based determination of bread size, texture, and other relevant information on production lines, enhancing quality control processes in future industrial settings (Fig. 5(c)). Using 1000 data sets for training, the system achieved remarkable performance. Both training and validation accuracies rapidly converged to high levels within 50 epochs, as shown in Fig. 5(d), demonstrating the efficiency of the learning process. The confusion matrix presented in Fig. 5(e) further validates the sensor's ability to provide data that clearly distinguishes between different textures and sizes of bread, achieving an overall accuracy of 92%. This demonstration of integration with basic machine learning tools emphasizes the SFP’s potential in various applications, from automated food quality assessment to advanced tactile sensing in robotic systems. The high-quality, multidimensional data from the SFP facilitates accurate identification and classification of grasped objects, showcasing its capability in handling real-world scenarios effectively without relying on visual information.

Herein, this study presents a smart finger patch designed for the index finger, integrating resistive and giant magnetoelastic sensors to achieve comprehensive tactile perception. The device successfully addresses the challenge of coupled perception of both state and process in dynamic tactile sensing, a crucial aspect for advancing human-machine interfaces in the Metaverse and VR/AR era. The SFP, comprising resistive CNT/PDMS films and magnetoelastic sensors, enables the detection of three key phases of finger action: bending state, dynamic contact behavior, and gripping force. The inward bending resistive sensor, featuring self-assembled microstructures, demonstrates excellent directional specificity and sensitivity. Concurrently, the magnetoelastic sensors exhibit remarkable responsiveness to various mechanical stimuli, including frequency, deformation magnitude, surface roughness, and material hardness. The system’s practical applicability is validated through a tactile-based bread type and condition recognition test, achieving an impressive 92% accuracy. This demonstration underscores the potential of the SFP in non-visual object recognition and quality assessment tasks. The developed electronic skin shows promise for a wide range of applications beyond VR/AR interfaces, including medical diagnostics, smart manufacturing, and industrial automation. While this study focuses on the index finger as a characteristic case due to its crucial role in tactile interactions, the technology demonstrates potential for extension to the entire hand. Its ability to provide rich, multidimensional tactile information opens new avenues for enhancing human-machine interactions in both virtual and real-world environments.