The Effects of Belief Elicitation in Visual Data Analysis: A Longitudinal Classroom Study

Recent studies have reported a shortcoming of visual exploratory data analysis (EDA) that can lead analysts to report spurious findings. These findings have prompted the advocacy of incorporating belief elicitation within the data analysis process. However, the results from these studies primarily drew from laboratory experiments, which can differ from real-world analysis contexts. In this article, we present outcomes from a longitudinal study with students enrolled in a university-level visual analytics course tackling the VAST Challenge. The students formed teams that were randomly assigned to the belief elicitation and non-belief elicitation conditions. Our study results indicate teams that underwent belief elicitation adopted an intentional approach, while teams in the non-belief elicitation condition reported greater diversity in findings, aligning with prior research. Surprisingly, teams from both conditions achieved equal success in solving the VAST Challenge, suggesting that analysts can incorporate belief elicitation strategically for different goals. We provide guidelines for incorporating belief elicitation into data analysis and teaching material for educators to include belief elicitation in visual analytics courses.