Quantifying Epistemic Relevance.

Area of Science:

• Cognitive Science
• Computational Linguistics
• Information Theory

Background:

• Relevance in information-seeking dialogue is often defined by progress toward resolving the question under discussion (QUD).
• Epistemic relevance quantifies this progress, but optimal measures for modeling human judgments remain unclear.
• Previous research has explored various information-theoretic approaches to quantify epistemic relevance.

Purpose of the Study:

• To experimentally evaluate and compare different quantitative measures of epistemic relevance.
• To determine which measure of epistemic relevance best predicts human relevance judgments for question-answer pairs.
• To investigate the role of first-order and second-order beliefs in assessing relevance.

Main Methods:

• Conducted an experimental evaluation of candidate epistemic relevance measures.
• Applied measures to question-answer pairs, focusing on polar questions.
• Compared models using first-order beliefs (point estimates) and second-order beliefs (probability distributions over probabilities).

Main Results:

• A Bayes factor measure, quantifying evidence provided by an answer, best predicted human relevance ratings for polar questions.
• This Bayes factor measure outperformed other information-theoretic measures like Kullback-Leibler divergence and change in entropy.
• Both first-order and second-order beliefs were found to play a role in predicting relevance judgments.

Conclusions:

• The Bayes factor provides a robust measure of epistemic relevance that aligns well with human judgments.
• Human relevance perception involves both epistemic factors (belief change) and non-epistemic factors.
• Further research is needed to fully characterize the multifaceted nature of relevance in dialogue.