Evaluating Visual Decision Support: How Does Preference Elicitation Shape Metric Sensitivity?

Area of Science:

• Information Visualization
• Human-Computer Interaction
• Decision Science

Background:

• Evaluating visualization effectiveness for decision-making is challenging due to a lack of objective metrics.
• Previous work used choice consistency with subjective preferences, but elicitation design impact is unclear.

Purpose of the Study:

• To investigate how different preference elicitation methods affect the sensitivity of decision quality metrics.
• To understand the influence of elicitation expressiveness on metrics used in comparative visualization studies.

Main Methods:

• A preregistered study with 548 participants compared parallel coordinates and tabular visualizations.
• Varied the expressiveness of preference elicitation methods across three conditions.
• Measured the sensitivity of choice consistency metrics to detect performance differences.

Main Results:

• A baseline condition with simple elicitation confirmed previous inconclusive findings on visualization performance.
• More expressive preference elicitation showed potential to increase metric sensitivity.
• Further increases in expressiveness did not yield additional benefits in sensitivity for this study.

Conclusions:

• Elicitation design significantly influences the sensitivity of decision-centric metrics in visualization studies.
• More expressive elicitation can enhance the detection of performance differences, informing metric development.
• Findings guide the interpretation of comparative visualization research and metric design.