Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Users.

Area of Science:

• Human-Computer Interaction
• Artificial Intelligence
• Assistive Technology

Background:

• General-purpose generative AI has established prompting and steering techniques.
• Assistive visual question answering (VQA) tools for blind users often lack interaction customization.
• Limited user control in VQA systems can be consequential for blind users relying on them for access.

Purpose of the Study:

• To investigate how blind users customize interactions with a conversational VQA system.
• To analyze prompting-based techniques used by blind users for VQA customization.
• To identify interaction design insights for improving VQA systems for blind users.

Main Methods:

• Recruited 11 blind users to customize a real-world conversational VQA system.
• Collected data from 418 interactions, user reflections, and post-study interviews.
• Analyzed prompting techniques adopted by participants, including self-developed methods.

Main Results:

• VQA interactions were lengthy, averaging 3 turns (up to 21), with short inputs and long responses.
• The state-of-the-art LLM-based system had limitations in verbosity control, spatial/temporal estimation, image framing, and camera guidance.
• Participants utilized prompt engineering to navigate and overcome system limitations.

Conclusions:

• Prompt engineering enables blind users to customize VQA interactions and mitigate system shortcomings.
• Customization techniques are crucial for enhancing the usability and accessibility of VQA systems for visually impaired users.
• The study provides a new dataset and actionable insights for designing more adaptive and user-centered VQA systems.