Related Experiment Videos
Inter-rater agreement in judging errors in diagnostic reasoning
Memoona Hasnain1, Hirotaka Onishi, Arthur S Elstein
1Department of Family Medicine, College of Medicine, University of Illinois at Chicago, 1919 W. Taylor Street, Chicago, IL 60612-7309, USA. memoona@uic.edu
Medical Education
|June 11, 2004
Summary
Clinicians using computerised decision support systems achieved good agreement in error categorization with explicit rules. However, the kappa coefficient may underestimate agreement when expected agreement is high.
Area of Science:
- Medical Informatics
- Clinical Decision Making
- Health Professions Education
Background:
- Inter-rater agreement is crucial for evaluating clinician performance.
- Accurate error categorization in the diagnostic process is essential for quality improvement.
Purpose of the Study:
- To assess inter-rater agreement in classifying diagnostic errors made by clinicians using computerised decision support systems (CDSS).
Main Methods:
- Developed eight error categories for diagnostic hypotheses and work-up plans.
- Two independent physician raters evaluated 54 patient work-ups from 2 cases.
- Calculated inter-rater agreement using the kappa coefficient.
Main Results:
- High agreement was observed across most error categories.
- Agreement decreased when raters relied on clinical judgment due to non-specific guidelines.
- The kappa coefficient indicated poor agreement when expected agreement was high and variability low.
Conclusions:
- Explicit scoring rules enhance clinician agreement in error categorization with CDSS.
- The kappa coefficient's limitations necessitate alternative agreement measures.
- Recommends using dual indices, similar to sensitivity and specificity, for robust agreement assessment.