Related Experiment Video
Updated: Nov 27, 2025
10:39
Qualitative and Quantitative Validation of Tools with Rating Scales Aimed at Assessing the Quality of University Service-Learning
Published on: August 29, 2025
708
Using the Data Agreement Criterion to Rank Experts' Beliefs
Duco Veen1, Diederick Stoel2, Naomi Schalken1
1Department of Methods and Statistics, Utrecht University, 3584 CH 14 Utrecht, The Netherlands.
Entropy (Basel, Switzerland)
|December 3, 2020
Summary
This study introduces a method to rank experts by evaluating their knowledge and uncertainty using probability distributions. This approach helps assess expert prediction accuracy and prior-data agreement, aiding decision-making.
Area of Science:
- Decision Sciences
- Statistical Modeling
- Expert Elicitation
Background:
- Expert beliefs represent current knowledge crucial for decision-making.
- Ranking experts by the merit of their beliefs is challenging.
- Quantifying expert knowledge and uncertainty is essential for reliable decision support.
Purpose of the Study:
- To develop a method for ranking experts based on their knowledge and uncertainty.
- To assess the accuracy of expert predictions and the appropriateness of their stated uncertainty.
- To evaluate the agreement between expert-defined priors and empirical data.
Main Methods:
- Experts specify knowledge as probability distributions.
- Utilize an extended Data Agreement Criterion to evaluate prior-data (dis)agreement.
- Compare expert priors using Bayesian statistical methods, including Bayes factors.
Main Results:
- The proposed method allows for the quantitative ranking of experts.
- It enables assessment of how well expert predictions align with new data.
- The approach facilitates comparison of expert validity and potential convergence with data.
Conclusions:
- Ranking experts by knowledge and uncertainty is feasible using probability distributions.
- The method provides a framework for evaluating expert judgment against data.
- This approach can inform decision-making by identifying the most reliable expert insights.
Related Concept Videos
01:20Kendall's Coefficient of Concordance
774
Kendall's Coefficient of Concordance (W), also known as Kendall's W, is a non-parametric statistical measure used to assess the agreement or concordance between multiple raters or judges when they rank a set of items. It is often used when you have ordinal data (ranks) and you want to see if there is consistency or consensus among the raters. It is widely applied in research areas such as psychology, medicine, and social sciences, where multiple judges are asked to rank or rate subjects...
774
01:02Ranks
353
Unlike parametric methods, nonparametric statistics are ideal for nominal and ordinal data, requiring fewer assumptions about the population's nature or distribution. This makes nonparametric methods easier to apply and interpret, as they do not depend on parameters like mean or standard deviation. One common approach in nonparametric analysis is to sort data according to a specific criterion. For instance, we might arrange weather data from hottest to coldest days in a month or rank cities...
353
01:13Accuracy and Errors in Hypothesis Testing
457
Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
457
01:02Quantifying and Rejecting Outliers: The Grubbs Test
3.3K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
3.3K
01:20Spearman's Rank Correlation Test
1.2K
Spearman's rank correlation test, also known as Spearman's rho, is a nonparametric method for assessing the strength and direction of association between two variables. This test is particularly valuable when the data distribution is unknown or when the assumption of normality does not hold. Named after the English psychologist and statistician Dr. Charles Edward Spearman, it serves as the nonparametric counterpart to Pearson's correlation coefficient.
Spearman's test calculates correlation by...
Spearman's test calculates correlation by...
1.2K
01:00Detection of Gross Error: The Q Test
6.7K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.7K