Probing the Visualization Literacy of Vision Language Models: The Good, the Bad, and the Ugly

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Vision Language Models (VLMs) show potential in chart comprehension.
• Previous research primarily assessed VLM response accuracy, neglecting internal reasoning processes.
• Understanding VLM visualization literacy requires exploring their decision-making mechanisms.

Purpose of the Study:

• To adapt and apply attention-guided class activation maps (AG-CAM) for visualizing VLM internal reasoning in chart comprehension.
• To compare the performance and internal reasoning of open-source and closed-source VLMs on chart-based tasks.
• To investigate VLM spatial and semantic reasoning capabilities during chart question-answering.

Main Methods:

• Adapted attention-guided class activation maps (AG-CAM) for early fusion VLM architectures.
• Evaluated four open-source (ChartGemma, Janus 1B/7B, LLaVA) and two closed-source (GPT-4o, Gemini) VLMs.
• Analyzed AG-CAM results to understand feature importance and model reasoning.

Main Results:

• ChartGemma, a 3B parameter VLM, outperformed other open-source models and matched closed-source models in performance.
• VLMs demonstrated spatial reasoning by localizing chart features and semantic reasoning by linking visual elements to data.
• AG-CAM visualization revealed insights into VLM decision-making processes for chart QA.

Conclusions:

• The adapted AG-CAM method provides a novel approach for analyzing VLM internal reasoning in chart comprehension.
• Open-source VLMs, particularly ChartGemma, show competitive performance and interpretable reasoning capabilities.
• This work advances transparent and reproducible research in AI visualization literacy and chart QA.