Re-imagining sensory substitution through gestural control: Point-To-Tell 2

Area of Science:

• Human-Computer Interaction
• Assistive Technology
• Computer Vision

Background:

• Sensory substitution devices (SSDs) convert environmental data for individuals who are blind or have low vision (pBLV).
• Existing SSDs often provide undifferentiated information, potentially causing cognitive overload and limiting user control.
• There is a need for more intuitive and selective assistive technologies for pBLV.

Purpose of the Study:

• To introduce and technically validate Point-To-Tell 2, a selective, gesture-controlled system for pBLV.
• To enhance information relevance and reduce cognitive overload in assistive devices.
• To connect spatial position and identity of objects through gestural control and proprioception.

Main Methods:

• Integration of a monocular camera with AI pipelines for depth estimation, hand pose tracking, and object detection/segmentation.
• Development of a system that identifies 3D pointing direction and announces object names and distances.
• Utilizing pointing-based gestural control for private and efficient information selection.

Main Results:

• High hand pose tracking accuracy demonstrated in controlled indoor environments.
• Reliable ray-casting and object selection achieved, even with challenges in object recognition at distance.
• Stable distance estimates at close range, with a noted systematic bias.

Conclusions:

• The Point-To-Tell 2 system technically validates an improved assistive technology approach.
• Focusing system feedback and leveraging concurrent hand proprioception can reduce cognitive load and enhance spatial comprehension.
• Future work includes user testing and feature expansion for diverse scenarios.