Charts-of-Thought: Enhancing LLM Visualization Literacy Through Structured Data Extraction

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Data Visualization

Background:

• Assessing Large Language Models' (LLMs) visualization literacy is crucial for their application in data interpretation.
• Existing prompting techniques may not fully leverage LLMs' potential for visual data analysis.

Purpose of the Study:

• To evaluate the visualization literacy of state-of-the-art LLMs.
• To introduce and assess the effectiveness of the Charts-of-Thought prompting technique.

Main Methods:

• Three LLMs (Claude-3.7-sonnet, GPT-4.5-preview, Gemini-2.0-pro) were tested on the Visualization Literacy Assessment Test (VLAT).
• A novel prompting technique, Charts-of-Thought, was developed to guide LLMs through systematic data extraction, verification, and analysis.
• Performance was compared between standard prompts and the Charts-of-Thought method.

Main Results:

• Claude-3.7-sonnet achieved a VLAT score of 50.17 using Charts-of-Thought, significantly exceeding the human baseline of 28.82.
• The Charts-of-Thought method improved LLM performance, with score increases of 21.8% for GPT-4.5, 9.4% for Gemini-2.0, and 13.5% for Claude-3.7.
• Claude-3.7-sonnet achieved 100% accuracy on several challenging chart types, demonstrating substantial gains in visualization interpretation.

Conclusions:

• Modern multimodal LLMs can exceed human performance in visualization literacy with appropriate analytical frameworks.
• Structured prompting strategies like Charts-of-Thought are vital for complex visual interpretation by LLMs.
• This method has implications for improving LLM capabilities and enhancing visualization accessibility.