Related Experiment Video
Updated: Oct 13, 2025
08:12
A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments
Published on: March 1, 2022
2.6K
Weighting schemes and incomplete data: A generalized Bayesian framework for chance-corrected interrater agreement
Rutger van Oest1, Jeffrey M Girard2
1Department of Marketing, BI Norwegian Business School.
Psychological Methods
|November 12, 2021
Summary
This study generalizes a framework for interrater agreement to handle various data types and missing information. The uniform prior coefficient generally performs best, especially with imbalanced category proportions and missing data.
Area of Science:
- Statistics
- Psychometrics
- Data Analysis
Background:
- Existing frameworks for interrater agreement often have limitations regarding data types and completeness.
- Van Oest (2019) proposed a framework for nominal categories and complete data.
Purpose of the Study:
- To generalize Van Oest's framework to accommodate nominal/ordinal categories and complete/incomplete data.
- To develop a comprehensive chance-corrected agreement coefficient.
Main Methods:
- Mathematical generalization of Van Oest's framework.
- Incorporation of Bayesian estimates for category proportions.
- Simulation studies to compare nested coefficients.
Main Results:
- The generalized coefficient accommodates weighting schemes, multiple raters/categories, and incomplete data.
- The uniform prior coefficient generally outperforms other coefficients, particularly with skewed category distributions.
- Scott's pi and Fleiss' kappa performance degrades with lenient weighting and missing data.
Conclusions:
- The generalized framework provides a flexible tool for assessing interrater agreement across diverse data scenarios.
- The uniform prior coefficient offers robust performance, especially in challenging data conditions.
- A new interpretation of weighted agreement coefficients as the probability of non-random correct classification is proposed.
Related Concept Videos
00:57Weighted Mean
5.7K
While taking the arithmetic, geometric, or harmonic mean of a sample data set, equal importance is assigned to all the data points. However, all the values may not always be equally important in some data sets. An intrinsic bias might make it more important to give more weightage to specific values over others.
For example, consider the number of goals scored in the matches of a tournament. While computing the average number of goals scored in the tournament, it may be more important to...
For example, consider the number of goals scored in the matches of a tournament. While computing the average number of goals scored in the tournament, it may be more important to...
5.7K
01:24One-Compartment Open Model: Wagner-Nelson and Loo Riegelman Method for ka Estimation
795
This lesson introduces two critical methods in pharmacokinetics, the Wagner-Nelson and Loo-Riegelman methods, used for estimating the absorption rate constant (ka) for drugs administered via non-intravenous routes. The Wagner-Nelson method relates ka to the plasma concentration derived from the slope of a semilog percent unabsorbed time plot. However, it is limited to drugs with one-compartment kinetics and can be impacted by factors like gastrointestinal motility or enzymatic degradation.
On...
On...
795
01:10Propagation of Uncertainty from Systematic Error
1.0K
The atomic mass of an element varies due to the relative ratio of its isotopes. A sample's relative proportion of oxygen isotopes influences its average atomic mass. For instance, if we were to measure the atomic mass of oxygen from a sample, the mass would be a weighted average of the isotopic masses of oxygen in that sample. Since a single sample is not likely to perfectly reflect the true atomic mass of oxygen for all the molecules of oxygen on Earth, the mass we obtain from this...
1.0K
01:16Statistical Inference Techniques in Hypothesis Testing: Parametric Versus Nonparametric Data
252
Statistical inference techniques, paramount in hypothesis testing, differentiate into two broad categories: parametric and nonparametric statistics.
Parametric statistics, as the name suggests, assumes that data follow a specific distribution, often a normal distribution. This assumption enables robust hypothesis testing and estimation. Parametric methods, like the Student's t-test or Goodness-of-fit test, are frequently employed in biostatistics due to their robustness. For instance,...
Parametric statistics, as the name suggests, assumes that data follow a specific distribution, often a normal distribution. This assumption enables robust hypothesis testing and estimation. Parametric methods, like the Student's t-test or Goodness-of-fit test, are frequently employed in biostatistics due to their robustness. For instance,...
252
01:19Interpretation of Confidence Intervals
7.3K
A confidence interval is a better estimate of the population than a point estimate, as it uses a range of values from a sample instead of a single value.
Confidence intervals have confidence coefficients that are crucial for their interpretation. The most common confidence coefficients are 0.90, 0.95, and 0.99, which can be written as percentages–90%, 95%, and 99%, respectively.
Suppose a person calculates a confidence interval with a confidence coefficient of 0.95. In that case, they can...
Confidence intervals have confidence coefficients that are crucial for their interpretation. The most common confidence coefficients are 0.90, 0.95, and 0.99, which can be written as percentages–90%, 95%, and 99%, respectively.
Suppose a person calculates a confidence interval with a confidence coefficient of 0.95. In that case, they can...
7.3K
01:13Accuracy and Errors in Hypothesis Testing
369
Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
369