Promoting insight-based evaluation of visualizations: from contest to benchmark repository

Area of Science:

• Information Visualization (InfoVis)
• Human-Computer Interaction (HCI)
• Data Science

Background:

• Information Visualization (InfoVis) is a growing field with ongoing questions about the maturity and optimal application of novel visualization techniques.
• While usability studies and controlled experiments offer insights, generalizing their findings remains challenging.
• Systematic development of benchmarks is crucial for comparing visualization techniques and identifying their performance under various conditions.

Purpose of the Study:

• To evaluate the effectiveness of novel visualization techniques through a series of information visualization contests.
• To establish benchmarks for comparing different visualization methods and understanding their strengths.
• To explore the potential of visualization contests as a method for advancing the field of InfoVis.

Main Methods:

• Organization and management of three information visualization contests at the IEEE InfoVis Symposia (2003, 2004, 2005).
• Contestants were tasked with reporting insights gained from data exploration using provided visualization tools.
• Archiving all contest materials in the InfoVis Benchmark Repository for future research.

Main Results:

• A summary of the state-of-the-art in information visualization evaluation was presented.
• The organization, results, and outcomes of the three InfoVis contests were detailed.
• Lessons learned from the contests and future directions for visualization competitions were discussed.

Conclusions:

• Information visualization contests provide a valuable framework for evaluating and comparing visualization techniques.
• The established benchmarks and archived materials facilitate objective comparison and advance the field.
• Visualization contests hold promise for the future development and standardization of InfoVis evaluation methods.