Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models

Area of Science:

• Human-Computer Interaction
• Assistive Technology
• Artificial Intelligence

Background:

• Blind individuals rely on screen readers for computer interaction.
• Heterogeneous application interfaces pose significant navigation challenges for screen reader users.
• Existing methods require memorizing numerous keyboard shortcuts and navigation patterns.

Purpose of the Study:

• To introduce Savant, a novel assistive technology designed to unify application interface interaction for blind screen reader users.
• To reduce the cognitive load and interaction burden associated with diverse graphical user interfaces.
• To enable uniform interaction across applications using natural language.

Main Methods:

• Development of Savant, an assistive technology leveraging large language models (LLMs).
• Implementation of natural language processing to interpret user commands.
• Automation of screen reader actions based on flexible, natural language prompts.
• User study involving 11 blind participants to evaluate Savant's effectiveness.

Main Results:

• Savant enables blind users to interact with any application interface through natural language.
• The system automates complex screen reader actions based on user commands.
• User commands do not require precise naming of interface elements.
• Significant improvements in interaction efficiency and usability were observed.

Conclusions:

• Savant offers a unified and efficient interaction method for blind screen reader users.
• LLM-powered assistive technology can effectively address challenges in navigating heterogeneous interfaces.
• The findings suggest a promising future for natural language-based assistive tools.