Opportunities for Human-AI Collaboration in Remote Sighted Assistance

Area of Science:

• Human-Computer Interaction
• Computer Vision
• Assistive Technology

Background:

• Remote sighted assistance (RSA) is a conversational technology aiding individuals with visual impairments (VI) using remote sighted agents for real-time navigation.
• Existing RSA systems face technical and navigational challenges for both agents and users.

Purpose of the Study:

• To comprehensively identify and understand the technical and navigational challenges in Remote Sighted Assistance (RSA).
• To explore the potential of computer vision (CV) and propose new directions for Human-CV collaboration to enhance RSA.

Main Methods:

• Conducted a literature review on Remote Sighted Assistance (RSA).
• Interviewed 12 users of RSA to gather insights on their experiences and challenges.
• Categorized technical challenges and identified real-world navigational scenarios.

Main Results:

• Identified four categories of technical challenges: user localization, environmental perception, information delivery, and network connectivity.
• Detailed 15 real-world navigational scenarios (8 outdoor, 7 indoor) highlighting user difficulties.
• Found that current CV technologies partially address challenges, but significant gaps remain.

Conclusions:

• Significant advancements in Human-CV collaboration are required to address the full spectrum of RSA challenges.
• Five emerging problems in Human-CV collaboration are proposed: blind-aware algorithms, robust localization, digital screen recognition, irregular text recognition, and trajectory prediction.
• Solving these problems will advance CV research and enable the next generation of RSA.