EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts.

Area of Science:

• Computer Science
• Artificial Intelligence
• Data Visualization

Background:

• Multimodal vision-language models (VLMs) show increasing performance on chart understanding tasks.
• Current benchmarks may not fully assess the visual reasoning required for comprehensive chart interpretation.

Purpose of the Study:

• Introduce ENCQA, a novel benchmark for evaluating visual reasoning in chart understanding.
• Systematically assess VLM capabilities across diverse visual encodings and analytical tasks.

Main Methods:

• Developed ENCQA with 2,076 synthetic question-answer pairs.
• Covered six visual encoding channels (position, length, area, color quantitative, color nominal, shape).
• Included eight analytical tasks (e.g., find extrema, retrieve value, correlate values).

Main Results:

• Evaluated 9 state-of-the-art VLMs on the ENCQA benchmark.
• Observed significant performance variations across different encodings and tasks.
• Found no consistent performance improvement with increased model size for many task-encoding combinations.

Conclusions:

• Current VLMs have specific visual reasoning gaps in chart understanding.
• Advancing chart interpretation requires targeted improvements beyond scaling model or dataset size.
• ENCQA provides a framework for identifying and addressing these specific visual reasoning deficiencies.