Do LLMs Have Visualization Literacy? An Evaluation on Modified Visualizations to Test Generalization in Data...

Area of Science:

• Artificial Intelligence
• Data Visualization
• Human-Computer Interaction

Background:

• Large Language Models (LLMs) demonstrate potential in generating chart descriptions and design suggestions.
• The capability of LLMs to evaluate data visualizations remains largely unexplored.
• Human data collection for visualization evaluation is time-consuming and costly, highlighting the need for automated solutions.

Purpose of the Study:

• To assess the visualization literacy of prominent LLMs, specifically OpenAI's GPT-4 and Google's Gemini.
• To establish benchmarks for evaluating LLM capabilities in understanding and interpreting data visualizations.
• To investigate the feasibility of using LLMs as evaluators in the visualization research process.

Main Methods:

• A modified 53-item Visualization Literacy Assessment Test (VLAT) was administered to GPT-4 and Gemini.
• LLM responses were analyzed to gauge their understanding of visual data.
• Performance was compared against existing human data from the VLAT.

Main Results:

• Both GPT-4 and Gemini demonstrated lower visualization literacy compared to the general public.
• LLMs exhibited a tendency to rely on their pre-existing knowledge base.
• The models struggled to interpret and utilize information directly presented in visualizations.

Conclusions:

• Current LLMs lack the necessary visualization literacy for effective use as evaluators in visualization research.
• Further development is required to enhance LLMs' ability to interpret visual data accurately.
• LLMs' reliance on prior knowledge over visual evidence limits their utility in data visualization assessment.