How to Ask What to Say?: Strategies for Evaluating Natural Language Interfaces for Data Visualization

Area of Science:

• Computer Science
• Human-Computer Interaction
• Data Visualization

Background:

• Natural Language Interfaces (NLIs) are increasingly used for data visualization.
• Evaluating the effectiveness of these NLIs presents unique challenges.
• Standardized evaluation methodologies are lacking.

Purpose of the Study:

• To discuss challenges and strategies for evaluating NLIs in data visualization.
• To provide guidance for researchers developing and assessing visualization NLIs.
• To identify areas for future research in NLI evaluation.

Main Methods:

• Review of prior studies on visualization NLI evaluation.
• Reflection on practical experiences in evaluating visualization NLIs.
• Analysis of three task framing strategies: Jeopardy-style facts, open-ended tasks, and target replication tasks.

Main Results:

• Different task framing strategies offer distinct benefits and considerations for NLI evaluation.
• Common challenges and pitfalls in NLI evaluation were identified.
• The importance of task design in eliciting meaningful performance data was highlighted.

Conclusions:

• Effective evaluation of visualization NLIs depends on strategic task framing.
• Future research should focus on developing new evaluation metrics and considering input modalities (spoken vs. typed).
• This work aims to guide researchers in conducting robust evaluations of visualization NLIs.