Re-imagining sensory substitution through gestural control: Point-To-Tell 2

Sensory substitution devices (SSDs) can convert environmental information into an accessible format for people who are blind or have low vision (pBLV). Yet, current SSDs often passively deliver all of the information available with limited user control, potentially leading to confusion and/or cognitive overload. To address this issue, this work proposes a selective, gesture-controlled system intended to improve information relevance and reduce cognitive overload.

We present Point-To-Tell 2, a system that enables pBLV to privately and efficiently select which information to convey through simple pointing-based gestural control. By integrating a monocular camera with AI-driven pipelines for depth estimation, hand pose tracking, and object detection/segmentation, the system identifies the users' 3D pointing direction and announces the names and distances of objects as they are pointed at, thereby connecting an object's spatial position and identity through hand proprioception.

Validation tests in controlled indoor environments show high hand pose tracking accuracy, ensuring reliable ray-casting and object selection despite declining object recognition at longer distances. Distance estimates are stable at close range, though a systematic bias is present.

This work introduces and technically validates an assistive system designed to improve the usability of assistive technologies by focusing system feedback-potentially reducing users' cognitive load and enhancing their spatial comprehension by leveraging concurrent hand proprioception. Future work will involve user testing and expanding system features to further enhance its practicality across more diverse scenarios.