Evaluating the evaluators: interrater reliability on EMT licensing examinations

Current methods of evaluating the technical competence of Michigan emergency medical technician (EMT) licensure candidates are subjective and potentially unreliable. Evaluators are required to attend a workshop before evaluating practical examination candidates. Despite the workshop, there is too much score variation and not enough observational consistency on the standardized examination.

To determine the level of rater reliability for evaluators of EMT practical examinations.

Data were collected from 104 licensed instructor-coordinators (ICs). Participants watched and scored two practical examinations simulated on videotape, one passing and one failing performance. Variation in student score and level of evaluator agreement concerning skill performance were determined. Michigan's basic EMT practical examination scoring instrument was used.

Nine basic EMT, 9 EMT specialist, and 86 EMT-paramedic ICs participated. Thirty-four percent had high school diplomas, 43% associate's degrees, 19% bachelor's degrees, 3% master's degrees, and 1% a doctoral degree. The ICs averaged 14 years of provider experience, and 6.35 years as an IC. The average score for scenario 1 (passing) was 86.4% (SD = 9.15, range = 53). An 80% score is required to pass. Nine ICs (9%) failed the student. Scenario 1 evaluator agreement was 79.4%. The average score for scenario 2 (failing) was 60.9% (SD = 11.57, range = 57). Seventeen ICs (18%) gave the student a passing score. Scenario 2 evaluator agreement was 67.8%. There was no significant difference in student scores based on evaluator level of education, licensure, or evaluator workshop attendance for either scenario.

Notable variation in scores given by evaluators for a single observed student, combined with low levels of evaluator agreement about skill performance, suggests that evaluators do not reliably rate student performance using the Michigan practical examination instrument.