Related Experiment Video
Updated: May 26, 2026
07:36
Mechanical Ventilation Boot Camp Curriculum
Published on: March 12, 2018
Interevaluator reliability of a mock paramedic practical examination
Melisa Martin1, Michael W Hubble, Marianne Hollis
1Emergency Medical Care Program, Western Carolina University, Cullowhee, North Carolina 28723, USA.
Prehospital Emergency Care
|January 11, 2012
Summary
Paramedic students
Area of Science:
- Medical Education
- Emergency Medical Services
- Psychomotor Skill Assessment
Background:
- Paramedic students require terminal competency assessment before national credentialing.
- Practical and oral examinations using skill sheets are common for evaluating psychomotor skills.
- Interevaluator reliability of these assessments is often unknown.
Purpose of the Study:
- To estimate the interevaluator reliability of paramedic skills during terminal competency testing.
- To assess variability in scoring among qualified evaluators.
Main Methods:
- Mock paramedic examinations were videotaped, featuring prespecified errors.
- Qualified evaluators scored performances using standardized skill sheets.
- Interevaluator variability was measured using standard deviation, range, and Krippendorff's alpha.
Main Results:
- High variability in total scores and critical errors was observed across evaluators and stations.
- Krippendorff's alpha coefficients indicated low reliability (ranging from 0.01 to 0.48).
- No correlation was found between total scores and evaluator demographics.
Conclusions:
- Qualified evaluators demonstrated high variability and low reliability when using skill sheets for terminal competency assessment.
- Further research is needed to establish true interevaluator reliability and optimal evaluator characteristics.
