A Review and Analysis of Evaluation Practices in VIS Domain Applications

Area of Science:

• Information Science
• Computer Science

Background:

• Evaluation practices in visualization and visual analytics (VIS) are crucial for assessing the effectiveness of VIS tools and methods.
• A comprehensive understanding of current evaluation trends in VIS domain applications is lacking.

Purpose of the Study:

• To review and analyze evaluation practices in visualization and visual analytics (VIS) domain application work.
• To establish a classification principle for evaluation practices and apply it to identify trends.
• To provide insights and highlight concerns for future evaluation in VIS research.

Main Methods:

• Systematic review and analysis of 140 papers from IEEE VIS conference (2018-2022).
• Development of a classification principle using Who, When, What, and How indicators (analysis methods, targets, scenarios, participant expertise, stages).
• Categorization of application domains and analysis of evaluation practices within them.

Main Results:

• A detailed classification principle for VIS evaluation practices was established.
• Prevailing characteristics and trends in evaluation practices across different application domains were identified.
• Variations in the usage of evaluation methods across different application domains were observed.

Conclusions:

• The study provides a framework for understanding and categorizing VIS evaluation practices.
• Insights and concerns regarding current evaluation methods are highlighted for future research.
• The findings aim to inform and guide subsequent studies in visualization and visual analytics evaluation.