Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Hubert's multi-rater kappa revisited.

Antonio Martín Andrés1, María Álvarez Hernández2

  • 1Bioestadística, Facultad de Medicina, Universidad de Granada, Granada, Spain.

The British Journal of Mathematical and Statistical Psychology
|May 7, 2019
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Comparison of predictive values with paired samples.

Journal of applied statistics·2026
Same author

Miettinen and Nurminen score statistics revisited.

Journal of biopharmaceutical statistics·2024
Same author

Two-tailed asymptotic inferences for the odds ratio in cross-sectional studies: evaluation of fifteen old and new methods of inference.

Journal of biopharmaceutical statistics·2020
Same author

Comments on "Two-tailed significance tests for 2 × 2 contingency tables: What is the alternative?" by Robin J Prescott Statistics in Medicine 2019;38:4264-4269.

Statistics in medicine·2020
Same author

Comment on "Efficient statistical inference for a parallel study with missing data by using an exact method" by Shan et al. (2019).

Journal of biopharmaceutical statistics·2019
Same author

Rejoinder to Mr. Peter J. Laud.

Journal of biopharmaceutical statistics·2018
Same journal

Connections between the basic local independence model and true and error models in the study of preference.

The British journal of mathematical and statistical psychology·2026
Same journal

Proficiency order invariance of MLE, MAP, EAP, and WLE in item response theory.

The British journal of mathematical and statistical psychology·2026
Same journal

Bias and precision in true-score estimation.

The British journal of mathematical and statistical psychology·2026
Same journal

Polychoric correlations under the assumption of elliptical latent traits.

The British journal of mathematical and statistical psychology·2026
Same journal

Regularized reduced rank regression for mixed predictor and response variables.

The British journal of mathematical and statistical psychology·2026
Same journal

A multiple-choice SDT model for cognitive diagnosis models.

The British journal of mathematical and statistical psychology·2026
See all related articles

This study addresses measuring agreement among multiple raters using kappa-type coefficients. It introduces Hubert

Area of Science:

  • Statistics
  • Psychometrics
  • Data Analysis

Background:

  • Measuring inter-rater reliability is crucial for consistent data collection across multiple observers.
  • Existing kappa-type coefficients have limitations, particularly when more than two raters are involved.
  • The interpretation of 'agreement' varies, leading to different coefficient definitions.

Purpose of the Study:

  • To provide a unified framework for kappa-type coefficients applicable to R > 2 raters.
  • To introduce Hubert's (nominal) and Schuster and Smith's (ordinal) kappa coefficients based on a strict agreement definition.
  • To derive formulae for the large-sample variances of these estimators for robust statistical inference.

Main Methods:

  • Utilized Hubert's interpretation of agreement: all raters must agree on a subject's category.
Keywords:
Cohen's kappaConger's kappaHubert's kappaSchuster and Smith's kappanominal agreementordinal agreement

Related Experiment Videos

  • Derived kappa coefficients for nominal (Hubert's kappa) and ordinal (Schuster and Smith's kappa) data.
  • Developed formulae for the large-sample variances of the estimated kappa coefficients.

    • Main Results:

    • Presented Hubert's and Schuster and Smith's kappa coefficients as extensions for multiple raters.
    • Provided variance formulae enabling statistical inference and comparison of different agreement measures.
    • Demonstrated that Schuster and Smith's ordinal kappa aligns with intraclass and concordance correlation coefficients under quadratic weighting.

    Conclusions:

    • Hubert's and Schuster and Smith's kappa coefficients offer robust measures of inter-rater agreement for R > 2 observers.
    • The derived variance formulae facilitate hypothesis testing and confidence interval construction for these coefficients.
    • The alignment of Schuster and Smith's kappa with correlation coefficients in the ordinal case provides a valuable link between different reliability metrics.