Quantifying Epistemic Relevance

In ordinary information-seeking dialogue, the relevance of an assertion seems to depend on how much progress it makes toward resolving the question under discussion (QUD). This kind of relevance-epistemic relevance-has been quantified in more than one way in prior work. However, there has been little investigation into which measures of epistemic relevance best model human judgments. We present the most comprehensive (to date) experimental evaluation of different candidate notions of epistemic relevance applied to question-answer pairs. We find that a measure based on the Bayes factor, which quantifies the amount of evidence provided by the answer, is the best predictor of relevance ratings for responses to polar questions, outperforming other information-theoretic measures based on Kullback-Leibler divergence and change in entropy. We compare models that use first-order beliefs (point estimates) and models that use second-order beliefs (probability density functions over probabilities). Our findings suggest that both first-order and second-order beliefs play a role in predicting introspective judgments. However, intuitions about relevance are not purely epistemic, meaning factors other than belief change are needed to fully characterize what it means to be relevant.