Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Resampling probability values for weighted kappa with multiple raters.

Paul W Mielke1, Kenneth J Berry, Janis E Johnston

  • 1Department of Statistics, Colorado State University, Fort Collins, CO 80523-1877, USA. mielke@lamar.colostate.edu

Psychological Reports
|June 24, 2008
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Spatial-temporal association of soil Pb and children's blood Pb in the Detroit Tri-County Area of Michigan (USA).

Environmental research·2020
Same author

The concurrent decline of soil lead and children's blood lead in New Orleans.

Proceedings of the National Academy of Sciences of the United States of America·2019
Same author

Spatiotemporal exposome dynamics of soil lead and children's blood lead pre- and ten years post-Hurricane Katrina: Lead and other metals on public and private properties in the city of New Orleans, Louisiana, U.S.A.

Environmental research·2017
Same author

Spatiotemporal dynamic transformations of soil lead and children's blood lead ten years after Hurricane Katrina: New grounds for primary prevention.

Environment international·2016
Same author

Evolving from reactive to proactive medicine: community lead (Pb) and clinical disparities in pre- and post-Katrina New Orleans.

International journal of environmental research and public health·2014
Same author

Environmental and health disparities in residential communities of New Orleans: the need for soil lead intervention to advance primary prevention.

Environment international·2012
Same journal

Continued Influence Effect: A Three-Dimensional Framework Shaping Practical and Theoretical Perspectives.

Psychological reports·2026
Same journal

ADHD Symptoms, Interpersonal Functioning, and Sexual Orientation in Undergraduate Adults.

Psychological reports·2026
Same journal

Emotion Regulation, Impulsivity, and Cluster B Personality Disorders.

Psychological reports·2026
Same journal

Physical Activity and Social Anxiety in Highly Sensitive Individuals: The Mediating Role of Cognitive Reappraisal.

Psychological reports·2026
Same journal

Development and Psychometric Evaluation of the Children's Prosocial Behaviors in a Preschool Setting (CPBPS).

Psychological reports·2026
Same journal

"Advances in Public & Social Psychology" Special Session of WPMH2026.

Psychological reports·2026
See all related articles

A novel resampling method estimates agreement for multiple raters using weighted kappa. This statistical procedure provides probability values for assessing inter-rater reliability in classification tasks.

Area of Science:

  • Statistics
  • Biostatistics
  • Psychometrics

Background:

  • Inter-rater reliability is crucial for consistent data collection and analysis.
  • Traditional agreement measures may not adequately account for chance agreement or the importance of disagreements.
  • Weighted kappa is a valuable metric for assessing agreement, especially when partial agreement is meaningful.

Purpose of the Study:

  • To introduce a new statistical procedure for computing weighted kappa with multiple raters.
  • To present a resampling technique for approximating probability values associated with multi-rater weighted kappa.
  • To demonstrate the practical application of this method using real-world classification data.

Main Methods:

  • Development of a novel computational procedure for weighted kappa.

Related Experiment Videos

  • Implementation of a resampling (e.g., bootstrapping) approach to estimate statistical significance.
  • Application of the procedure to a dataset with classifications from three independent raters.

    • Main Results:

    • The described procedure successfully computes weighted kappa for multiple raters.
    • The resampling method provides approximate probability values, enabling statistical inference.
    • The example analysis demonstrates the utility of the method in evaluating classification consistency.

    Conclusions:

    • The new procedure offers a robust method for assessing inter-rater reliability with weighted kappa in multi-rater scenarios.
    • The resampling technique enhances the interpretability of weighted kappa by providing probability estimates.
    • This approach is valuable for researchers and practitioners dealing with subjective classifications and multiple evaluators.