Assessing the assessment in emergency care training.

Area of Science:

• Medical Education
• Healthcare Professional Training
• Emergency Medicine Assessment

Background:

• Millions of healthcare professionals annually attend emergency care courses.
• Limited evidence exists on the quality of assessments used in these high-stakes courses.
• Evaluating assessment methods is crucial for patient safety and resource allocation.

Purpose of the Study:

• To evaluate the validity and reliability of common assessment formats for emergency care skills.
• To compare the psychometric properties of checklists, competency scales, and global performance scales.
• To provide recommendations for improving high-stakes skill assessment in emergency medicine.

Main Methods:

• Psychometric analyses were performed on data from 144 residents and 12 raters.
• Assessment instruments included checklists, a 9-item competency scale, and a global performance scale.
• Subgroup analysis involved videotaping and re-assessment by multiple raters to determine inter-rater reliability.

Main Results:

• Checklists demonstrated limited validity and poor inter-rater reliability (ICC=.30-.39).
• The competency scale showed good construct validity (clinical and communication subscales) with high internal consistency (α=.93/.91/.86).
• Inter-rater reliability was moderate for the clinical competency subscale (.49) and global performance scale (.50), but poor for the communication subscale (.27).

Conclusions:

• Checklists are inadequate for valid and reliable emergency skills assessment.
• Clinical competency and global performance scales offer better validity and moderate reliability.
• Increasing the number of raters significantly improves assessment reliability, with recommendations for optimal rater numbers for each instrument.