Evaluating Visual Decision Support: How Does Preference Elicitation Shape Metric Sensitivity?

Supporting decision-making is a central goal in visualization research, yet evaluating how effectively visualizations aid decision tasks remains difficult. Objective metrics for decision quality as a baseline for comparative evaluations are rare, with previous work proposing the consistency between a decision made and self-reported subjective preferences. However, the impact of preference elicitation design on such metrics is not well understood. Our focus is therefore on understanding how elicitation methods shape the sensitivity of decision quality metrics, not on comparing visualization techniques themselves. We report a preregistered study with 548 participants examining how varying the expressiveness of preference elicitation affects the sensitivity of choice consistency metrics when comparing parallel coordinates and tabular visualizations. Our baseline condition replicates a previous study using simple preference elicitation with significantly more participants and confirms its inconclusive findings that suggest a comparable performance of parallel coordinates and tabular visualization for the tested decision task. We further test two additional conditions to investigate if more expressive preference elicitation increases the metric's sensitivity. Our results suggest that more expressive elicitation may improve sensitivity in detecting performance differences, but further increases offer no clear additional benefit in our study. We observed these sensitivity differences even though the experimental setup, with its comparable baseline performances, was not optimized for detecting visualization performance differences. Our results provide initial evidence that elicitation design can affect the informational value of decision-centric visualization studies, informing both the development of metrics and the interpretation of comparative studies.