Write, Rank, or Rate: Comparing Methods for Studying Visualization Affordances

Area of Science:

• Data Visualization
• Human-Computer Interaction
• Cognitive Psychology

Background:

• Understanding how specific design choices in information visualizations influence reader interpretation is crucial.
• Traditional methods for analyzing visualization affordances often rely on labor-intensive, crowdsourced studies generating extensive free-response data.

Purpose of the Study:

• To explore and compare scalable research methodologies for assessing visualization affordances.
• To evaluate the effectiveness of different elicitation methods in capturing reader interpretations of various chart types.

Main Methods:

• Tested four human-subject study methods: free response, visualization ranking, conclusion ranking, and salience rating.
• Compared these methods' ability to elicit interpretations for line charts, dot plots, and heatmaps.
• Investigated the use of GPT-4o as a large language model (LLM) proxy for human participants.

Main Results:

• No single method fully replicated free-response conclusions, but combined ranking and rating methods served as effective broad-scale proxies.
• Ranking methods showed bias towards specific chart types and presented conclusions.
• Salience rating did not capture nuanced variations between chart types.
• GPT-4o performed best as a proxy for salience rating but had limitations in other areas.

Conclusions:

• Combinations of elicitation methods can provide scalable insights into visualization affordances.
• Participant and model biases need careful consideration when selecting methodologies.
• The choice and combination of methods, including LLMs, significantly impact the observed visualization affordances and require careful evaluation of trade-offs.