Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Measuring interrater reliability among multiple raters: an example of methods for nominal data.

K L Posner1, P D Sampson, R A Caplan

  • 1Department of Anesthesiology, University of Washington, Seattle 98195.

Statistics in Medicine
|September 1, 1990
PubMed
Summary

This study evaluates methods for measuring inter-rater reliability with nominal data, crucial for consistent assessments in research. It proposes new estimators for stratified sampling and analyzes anaesthesiologist ratings for care appropriateness.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Observation of tt[over ¯] Production in Pb+Pb Collisions at sqrt[s_{NN}]=5.02  TeV with the ATLAS Detector.

Physical review letters·2025
Same author

Search for Dark Matter Produced in Association with a Dark Higgs Boson in the bb[over ¯] Final State Using pp Collisions at sqrt[s]=13  TeV with the ATLAS Detector.

Physical review letters·2025
Same author

Search for Magnetic Monopole Pair Production in Ultraperipheral Pb+Pb Collisions at sqrt[s_{NN}]=5.36  TeV with the ATLAS Detector at the LHC.

Physical review letters·2025
Same author

Simultaneous Unbinned Differential Cross-Section Measurement of Twenty-Four Z+jets Kinematic Observables with the ATLAS Detector.

Physical review letters·2025
Same author

Disentangling Sources of Momentum Fluctuations in Xe+Xe and Pb+Pb Collisions with the ATLAS Detector.

Physical review letters·2025
Same author

Search for Light Long-Lived Particles in pp Collisions at sqrt[s]=13  TeV Using Displaced Vertices in the ATLAS Inner Detector.

Physical review letters·2024

Area of Science:

  • Medical Statistics
  • Health Services Research
  • Anesthesiology

Background:

  • Accurate measurement of inter-rater reliability is essential for the validity of studies involving subjective assessments.
  • Existing methods for nominal data may not adequately address complex sampling designs or individual rater performance.

Purpose of the Study:

  • To review and critique existing approaches for measuring inter-rater reliability with nominal data.
  • To introduce modified estimators suitable for stratified sampling frames.
  • To assess both group and individual rater reliability.

Main Methods:

  • Review of statistical approaches for inter-rater reliability.
  • Development of modified kappa-like statistics for stratified samples.
  • Application of the jackknife method for standard error computation.

Related Experiment Videos

  • Analysis of independent ratings from 42 anaesthesiologists on 48 case histories.

    • Main Results:

    • The paper critiques current reliability measurement techniques for nominal data.
    • Modified estimators are proposed for stratified sampling scenarios.
    • The jackknife method is used to interpret computed standard errors.
    • Example analysis demonstrates the application to anaesthesiologist appropriateness ratings.

    Conclusions:

    • The study provides enhanced methods for assessing inter-rater reliability in complex settings.
    • The proposed techniques improve the measurement of reliability for nominal data, particularly with stratified sampling.
    • The findings are relevant for improving the quality and consistency of expert judgments in healthcare research.