VisEval: A Benchmark for Data Visualization in the Era of Large Language Models

Area of Science:

• Computer Science
• Data Visualization
• Artificial Intelligence

Background:

• Natural language to visualization (NL2VIS) is crucial for visual data analysis but technically demanding.
• Large language models (LLMs) show potential for NL2VIS, yet lack standardized evaluation benchmarks.
• Existing NL2VIS methods require complex, low-level implementations in natural language processing and visualization design.

Purpose of the Study:

• To address the need for a comprehensive benchmark for evaluating LLMs in NL2VIS tasks.
• To introduce VisEval, a novel benchmark comprising a large-scale dataset and automated evaluation methodology.
• To provide reliable insights into the capabilities and limitations of current LLMs for visualization generation.

Main Methods:

• Developed a high-quality, large-scale dataset with 2,524 queries across 146 databases and ground truth labels.
• Advocated for a comprehensive automated evaluation methodology assessing validity, legality, and readability of generated visualizations.
• Utilized heterogeneous checkers for systematic issue detection to ensure reliable evaluation outcomes.

Main Results:

• The VisEval benchmark was applied to several state-of-the-art LLMs.
• Evaluations revealed significant challenges in current LLMs' ability to generate accurate and effective visualizations from natural language.
• The study identified key areas for improvement in LLM-based NL2VIS systems.

Conclusions:

• VisEval provides a robust framework for benchmarking NL2VIS capabilities of LLMs.
• The findings highlight the necessity for further research and development to enhance LLM performance in visualization generation.
• This work offers essential insights for advancing the field of automated visual data analysis.