An Empirical Evaluation of the GPT-4 Multimodal Language Model on Visualization Literacy Tasks.

Area of Science:

• Computer Science
• Data Visualization
• Artificial Intelligence

Background:

• Large Language Models (LLMs) with multimodal capabilities, such as GPT-4, present new opportunities for advancing visualization research.
• The visual interpretation and data reading capabilities of these models remain largely unexplored.
• Assessing the visualization literacy of LLMs is crucial for understanding their potential and limitations.

Purpose of the Study:

• To evaluate the visualization literacy of the GPT-4 multimodal LLM.
• To assess its ability to interpret visually represented data and understand visualization principles.
• To identify the strengths and weaknesses of LLMs in the context of data visualization.

Main Methods:

• Developed and utilized a suite of task sets to assess GPT-4's visualization literacy.
• Task sets were based on established research in automated chart question answering and human visualization literacy.
• Evaluated GPT-4's performance on tasks including trend recognition, value retrieval, and color distinction.

Main Results:

• GPT-4 demonstrated proficiency in recognizing trends and extreme values in visualizations.
• The model showed an understanding of some visualization design best-practices.
• GPT-4 encountered difficulties with precise value retrieval without the original dataset and struggled with reliable color differentiation.
• Inconsistent performance and occasional hallucinations were observed.

Conclusions:

• GPT-4 exhibits potential for certain visualization tasks but has significant limitations.
• Challenges in data retrieval, color perception, and consistency need to be addressed for effective integration into visualization research.
• Further research is needed to explore the utility and refine the capabilities of LLMs in the field of data visualization.