TouchWGNN: spatio-temporal tactile perception for multimodal dexterous manipulation

Dexterous in-hand manipulation requires robotic hands to estimate object-state reliably under frequent occlusions, contact-rich interactions, and fast dynamics. Tactile sensing provides high-frequency, contact-specific feedback, although extracting useful representations from raw tactile signals and integrating them with vision and proprioception remains challenging. In this article, we present TouchWGNN, a multimodal dexterous manipulation framework that explicitly models tactile signals as a spatio-temporal graph. We first develop a low-cost distributed tactile sensor array for a five-fingered robotic hand, enabling real-time acquisition of normal forces from 113 sensing points distributed across key contact regions. We then construct a tactile graph in which taxels (or active contact points) are nodes with force features and 3D coordinates and edges encode spatial proximity and local force variation. A graph-based spatial encoder captures instantaneous contact geometry, and a temporal module models its evolution over time to refine object-state estimates. Finally, we fuse the tactile estimate with vision (point-cloud-based pose) and proprioception (joint states) for policy learning with reinforcement learning. Experiments on two in-hand manipulation tasks, cube reorientation and Baoding ball swapping, demonstrate that integrating raw tactile feedback with vision and proprioception improves manipulation performance compared with unimodal baselines.