Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

A method for measuring interrater agreement on checklists.

J M Sinacore1, K J Connell, A J Olthoff

  • 1University of Illinois at Chicago, Department of Family Medicine 60612-7248, USA. Sinacore@uic.edu

Evaluation & the Health Professions
|November 11, 1999
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

New draft rejection study refutes shocking national health.

Medical economics·2010
Same author

Statistics and the nation's health.

The Medical annals of the District of Columbia·2010
Same author

Simple directionally sensitive two-component laser Doppler anemometer.

Applied optics·2010
Same author

CORRELATION OF THE FINE STRUCTURE OF THE GASTRIC PARIETAL CELL (DOG) WITH FUNCTIONAL ACTIVITY OF THE STOMACH.

The Journal of biophysical and biochemical cytology·2009
Same author

Hemodynamic measurements in human arterial casts, and their correlation with histology and luminal area.

Journal of biomechanical engineering·2009
Same author

Association of educational attainment with HIV risk in African American active injection drug users.

AIDS care·2006
Same journal

The "Twilight Zone" Is a Danger Zone: Why the Occupational-Clinical Divide in Burnout Assessment Is a False Dichotomy.

Evaluation & the health professions·2026
Same journal

Evaluating Equity in AI-Supported Functional Assessment: Agreement Between Clinician Judgment and Digital Metrics in Stroke Rehabilitation.

Evaluation & the health professions·2026
Same journal

Psychometric Properties of the Arabic Version of the PROMIS Sleep Disturbance 8b Short Form Among Nurses.

Evaluation & the health professions·2026
Same journal

Commentary: Systemic Inequities in Japan's Technical Intern Training Program (TITP): Health, Labor, and Legal Vulnerabilities of Foreign Trainees.

Evaluation & the health professions·2026
Same journal

Application of Patient-Reported Outcome Measurements in Traditional Chinese Medicine Clinical Trials for Musculoskeletal Disorders in China: A Registry-Based Analysis.

Evaluation & the health professions·2026
Same journal

Divergent Socioeconomic Pathways to Biologically Uncontrolled Diabetes by Gender: A Bayesian Analysis of NHANES 2021-2023.

Evaluation & the health professions·2026
See all related articles

This study introduces a new method to measure interrater agreement using a single score, phi, derived from checklist configurations. Statistical simulation is used to determine chance agreement and significance, proving more robust than binomial distribution for dichotomous checklist data.

Area of Science:

  • Medical Education
  • Biostatistics
  • Psychometrics

Background:

  • Accurate measurement of interrater agreement is crucial for research reliability.
  • Existing methods may not adequately address dichotomous checklist data.
  • A need exists for a robust statistical approach to quantify checklist agreement.

Purpose of the Study:

  • To present a novel method for measuring interrater agreement on checklists.
  • To introduce the phi coefficient for quantifying agreement based on rater configurations.
  • To establish a simulation-based approach for determining chance agreement and statistical significance.

Main Methods:

  • Developed a method calculating a single agreement score (phi) from checklist responses.
  • Utilized statistical simulation to determine expected chance agreement.

Related Experiment Videos

  • Assessed the statistical significance of the phi coefficient.
  • Demonstrated the methodology using a medical education study.

    • Main Results:

    • The phi coefficient provides a single, comprehensive measure of interrater agreement.
    • Statistical simulation effectively determines chance agreement and significance for dichotomous checklist data.
    • The binomial distribution is inappropriate for testing the significance of phi in this context.

    Conclusions:

    • The proposed phi methodology offers a reliable way to measure interrater agreement on checklists.
    • Statistical simulation is a valid and necessary approach for assessing the significance of checklist agreement.
    • This method enhances the rigor of studies employing checklists, particularly in medical education.