Search research articles

ABOUT JoVE

Overview Leadership Blog JoVE Help Center

AUTHORS

Publishing Process Editorial Board Scope & Policies Peer Review FAQ Submit

LIBRARIANS

Testimonials Subscriptions Access Resources Library Advisory Board FAQ

RESEARCH

JoVE Journal Methods Collections JoVE Encyclopedia of Experiments Archive

EDUCATION

JoVE Core JoVE Business JoVE Science Education JoVE Lab Manual Faculty Resource Center Faculty Site

Terms & Conditions of Use

Related Concept Videos

One-Compartment Open Model: Wagner-Nelson and Loo Riegelman Method for ka Estimation

One-Compartment Open Model: Wagner-Nelson and Loo Riegelman Method for k_a Estimation

This lesson introduces two critical methods in pharmacokinetics, the Wagner-Nelson and Loo-Riegelman methods, used for estimating the absorption rate constant (ka) for drugs administered via non-intravenous routes. The Wagner-Nelson method relates ka to the plasma concentration derived from the slope of a semilog percent unabsorbed time plot. However, it is limited to drugs with one-compartment kinetics and can be impacted by factors like gastrointestinal motility or enzymatic degradation.
On...

Expected Frequencies in Goodness-of-Fit Tests

Expected Frequencies in Goodness-of-Fit Tests

A goodness-of-fit test is conducted to determine whether the observed frequency values are statistically similar to the frequencies expected for the dataset. Suppose the expected frequencies for a dataset are equal such as when predicting the frequency of any number appearing when casting a die. In that case, the expected frequency is the ratio of the total number of observations (n) to the number of categories (k).

Kendall's Coefficient of Concordance

Kendall's Coefficient of Concordance

Kendall's Coefficient of Concordance (W), also known as Kendall's W, is a non-parametric statistical measure used to assess the agreement or concordance between multiple raters or judges when they rank a set of items. It is often used when you have ordinal data (ranks) and you want to see if there is consistency or consensus among the raters. It is widely applied in research areas such as psychology, medicine, and social sciences, where multiple judges are asked to rank or rate subjects...

Accuracy and Errors in Hypothesis Testing

Accuracy and Errors in Hypothesis Testing

Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...

Confidence Interval for Estimating Population Mean

Confidence Interval for Estimating Population Mean

A point estimate of the population mean is obtained from a single sample. Such a point estimate does not represent a population well because it needs to account for variability in the population. Single point estimate can also be biased despite the sample being selected randomly. Thus, a point estimate is often unreliable. A confidence interval is needed to reduce this unreliability.
A confidence interval for the mean is a range of values that provides an estimate of the population mean. As the...

Friedman Two-way Analysis of Variance by Ranks

Friedman Two-way Analysis of Variance by Ranks

Friedman's Two-Way Analysis of Variance by Ranks is a nonparametric test designed to identify differences across multiple test attempts when traditional assumptions of normality and equal variances do not apply. Unlike conventional ANOVA, which requires normally distributed data with equal variances, Friedman's test is ideal for ordinal or non-normally distributed data, making it particularly useful for analyzing dependent samples, such as matched subjects over time or repeated measures...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by

Same author

Between Patterns and Predictions: Interpretable Latent EEG Representations for Clinical Insights.

medRxiv : the preprint server for health sciences·2026

Same author

BCGNet: an AI model trained on 600 K hours of sleep data for a novel under-pillow contactless monitoring device.

NPJ digital medicine·2026

Same author

Association between Interictal Spike Rate and Seizure Frequency in a Large Epilepsy Cohort.

medRxiv : the preprint server for health sciences·2026

Same author

Automated Prediction of Glasgow Coma Scale Scores From Unstructured Electronic Health Records Using Natural Language Processing: Development and Validation Study.

Journal of medical Internet research·2026

Same author

Quantification of Ventilatory Control in Sleep Apnea: From Physiological Insight to Computable Loop Gain.

Sleep·2026

Same author

Determinants of Delayed Recovery of Consciousness After Analgosedation Discontinuation in the ICU: Insights From Patients With COVID-19 Hypoxemic Respiratory Failure.

Critical care medicine·2026

Same journal

Regression Modeling of Individual-Patient Correlated Discrete Outcomes with Applications to Cancer Pain Ratings.

Open journal of statistics·2022

Same journal

Modeling Individual Patient Count/Rate Data over Time with Applications to Cancer Pain Flares and Cancer Pain Medication Usage.

Open journal of statistics·2022

Same journal

Constructing Statistical Intervals for Small Area Estimates Based on Generalized Linear Mixed Model in Health Surveys.

Open journal of statistics·2022

Same journal

A Co-Evolution Model for Dynamic Social Network and Behavior.

Open journal of statistics·2021

Same journal

Statistical Assessment of Neighborhood Socioeconomic Deprivation Environment in Spatial Epidemiologic Studies.

Open journal of statistics·2016

Same journal

Cusp Catastrophe Polynomial Model: Power and Sample Size Estimation.

Open journal of statistics·2016

See all related articles

Search research articles

Related Experiment Video

Updated: May 12, 2025

Lexical Decision Task for Studying Written Word Recognition in Adults with and without Dementia or Mild Cognitive Impairment

Lexical Decision Task for Studying Written Word Recognition in Adults with and without Dementia or Mild Cognitive Impairment

Published on: June 25, 2019

Interrater Reliability Estimation via Maximum Likelihood for Gwet's Chance Agreement Model.

Alek M Westover¹, Tara M Westover², M Brandon Westover²

¹Massachusetts Institute of Technology, Boston, MA, USA.

Open Journal of Statistics

|May 9, 2025

Summary

This summary is machine-generated.

This study introduces the maximum likelihood kappa (), an unbiased estimator for interrater reliability (IRR). It corrects biases in existing chance agreement models, improving the accuracy of IRR statistics.

Keywords:

Agreement Interrater Reliability Kappa Reliability

More Related Videos

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

Published on: March 1, 2022

Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits

Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits

Published on: September 27, 2019

Related Experiment Videos

Last Updated: May 12, 2025

Lexical Decision Task for Studying Written Word Recognition in Adults with and without Dementia or Mild Cognitive Impairment

Lexical Decision Task for Studying Written Word Recognition in Adults with and without Dementia or Mild Cognitive Impairment

Published on: June 25, 2019

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

Published on: March 1, 2022

Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits

Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits

Published on: September 27, 2019

Area of Science:

Statistics
Psychometrics
Data Science

Background:

Interrater reliability (IRR) quantifies agreement between observers.
Cohen's kappa is a common IRR statistic but has limitations.
Gwet's agreement statistic offers an alternative but has its own biases.

Purpose of the Study:

To address limitations in existing IRR statistics.
To develop an unbiased estimator for chance agreement.
To introduce the maximum likelihood kappa () statistic.

Main Methods:

Derivation of a maximum likelihood estimator for the occasional guessing model.
Identification of chance agreement probability with the observed disagreement rate.
Development of the maximum likelihood kappa () statistic.

Main Results:

Gwet's chance agreement formula is biased at intermediate agreement levels.
The maximum likelihood kappa () provides an unbiased IRR estimator.
Chance agreement probability equals the observed disagreement rate under the occasional guessing model.

Conclusions:

The maximum likelihood kappa () offers a theoretically sound IRR measure.
This new statistic overcomes limitations of Cohen's kappa and Gwet's statistic.
An unbiased IRR estimator is crucial for reliable analysis of rater judgments.