Exploring Multimodal Prompt for Visualization Authoring With Large Language Models

Area of Science:

• Human-Computer Interaction
• Artificial Intelligence
• Data Visualization

Background:

• Large language models (LLMs) show promise for automating visualization creation via natural language.
• Text-based prompting for LLMs in visualization authoring lacks precision, leading to errors and inefficiencies.

Purpose of the Study:

• To investigate LLM interpretation of ambiguous text prompts for visualization.
• To introduce and evaluate multimodal prompting (text, visual, direct manipulation) to improve visualization authoring.
• To understand the conditions causing LLM misinterpretation of user intent in visualization tasks.

Main Methods:

• Empirical study on LLM interpretation of text prompts in visualization authoring.
• Design and implementation of VisPilot, a system supporting multimodal prompts (text, sketches, direct manipulation).
• Evaluation of VisPilot via controlled user studies and expert assessments.

Main Results:

• Multimodal prompts significantly improve the clarity of user intent for LLMs in visualization authoring.
• Users effectively communicate spatial constraints, local references, and design preferences using multimodal prompts.
• Task efficiency with multimodal prompts is comparable to text-only prompting.

Conclusions:

• Multimodal prompting complements text-based methods, enhancing LLM-driven visualization authoring.
• VisPilot demonstrates the effectiveness of combining text, sketches, and direct manipulation for intuitive visualization creation.
• Findings offer design implications for future human-AI collaborative authoring systems.