MapIO: A Gestural and Conversational Interface for Tactile Maps

Area of Science:

• Human-Computer Interaction
• Assistive Technology
• Artificial Intelligence

Background:

• Tactile maps offer spatial information for the blind and visually impaired but have data limitations.
• Digitally augmented tactile maps use audio feedback to enhance tactile information.
• Existing systems lack dynamic, question-answering capabilities for tactile map exploration.

Purpose of the Study:

• To explore an embodied interaction paradigm augmenting tactile maps with Large Language Models (LLMs).
• To enable users to ask arbitrary questions and receive audio answers about map content.
• To evaluate the usability and user experience of this novel system.

Main Methods:

• Developed a system integrating LLMs for conversational interaction with tactile maps.
• Engineered LLM prompts to ensure reliable and relevant answers.
• Conducted a user study with 10 participants to assess interaction, usability, and user experience.

Main Results:

• Users can ask diverse questions about tactile map elements.
• The LLM-powered system provides relevant audio feedback.
• Initial evaluation indicates potential for improved spatial understanding and user engagement.

Conclusions:

• Conversational AI augmentation offers a promising new way to interact with tactile maps.
• This approach enhances accessibility and information retrieval for visually impaired users.
• Further research can refine LLM integration for richer, more intuitive map interactions.