Direct Observation Assessment of Milestones: Problems with Reliability.

Area of Science:

• Medical Education
• Emergency Medicine Training
• Assessment and Evaluation

Background:

• Emergency medicine (EM) milestones are essential for evaluating resident progress.
• Existing milestone assessment tools lack standardization and sufficient validity evidence.
• Concerns exist regarding the accurate measurement of intended resident competencies.

Purpose of the Study:

• To design and validate a direct observation instrument for assessing EM residents.
• To gather validity and reliability evidence for the new assessment tool.
• To provide further evidence for the accurate measurement capabilities of EM milestones.

Main Methods:

• A multi-center, prospective, observational validity study was conducted.
• The Critical Care Direct Observation Tool (CDOT) was developed using a modified Delphi method.
• Faculty raters assessed simulated resident performances using two CDOT versions; reliability was measured using Fleiss' kappa and interclass correlations.

Main Results:

• Two versions of the CDOT were piloted: one with global rating scales and one with a checklist format.
• Despite rater experience, significant variability in scoring occurred, resulting in poor inter-rater reliability.
• Fleiss' kappa values for all items on both CDOT versions were near zero.

Conclusions:

• The validity and reliability of current EM milestone assessment tools remain undetermined.
• This study highlights low inter-rater reliability in a rigorously developed direct observation instrument.
• Potential sources of error (rater and instrument) necessitate further investigation for EM milestone assessment reliability.