Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Evaluating the evaluators: interrater reliability on EMT licensing examinations

W Snyder1, S Smit

  • 1Mercy Memorial Hospital, Monroe, MI 48162, USA. snyder@mailgw-vtc.oto.med.umich.edu

Prehospital Emergency Care
|September 16, 1998
PubMed
Summary

Evaluators show significant score variation and low agreement when assessing emergency medical technician (EMT) practical exams. Current methods lack reliability for evaluating technical competence in EMT candidates.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Momentum-dependent scaling exponents of nodal self-energies measured in strange metal cuprates and modelled using semi-holography.

Nature communications·2024
Same author

Bayesian evaluation of three serological tests for the diagnosis of bovine brucellosis in Bangladesh.

Epidemiology and infection·2019
Same author

Acute administration of roflumilast enhances immediate recall of verbal word memory in healthy young adults.

Neuropharmacology·2017
Same author

Detection of viral RNA by RT-PCR from serum for molecular diagnosis of measles.

African journal of medicine and medical sciences·2011
Same author

Diagnostic evaluation of cancer antigens RAK .1. Cervical and ovarian cancer.

International journal of oncology·2011
Same author

Functional consequences of perturbing polyamine metabolism in the malaria parasite, Plasmodium falciparum.

Amino acids·2009

Area of Science:

  • Medical Education
  • Emergency Medical Services
  • Healthcare Professional Assessment

Background:

  • Current emergency medical technician (EMT) technical competence evaluations in Michigan are subjective and unreliable.
  • Existing practical examination scoring lacks observational consistency despite evaluator workshops.
  • There is a need for more dependable methods to assess EMT licensure candidates.

Purpose of the Study:

  • To determine the level of rater reliability among evaluators of EMT practical examinations.
  • To assess the consistency and agreement in scoring EMT practical skills.
  • To identify potential issues with the current Michigan EMT practical examination instrument.

Main Methods:

  • 104 licensed instructor-coordinators (ICs) watched and scored two videotaped practical EMT examinations (one passing, one failing).

Related Experiment Videos

  • Evaluator agreement and score variation were analyzed using Michigan's basic EMT practical examination scoring instrument.
  • Data on evaluator education, experience, and workshop attendance were collected.

    • Main Results:

    • Evaluator agreement for the passing scenario was 79.4%, with 9% incorrectly failing the student.
    • Evaluator agreement for the failing scenario was 67.8%, with 18% incorrectly passing the student.
    • No significant differences in scores were found based on evaluator education, licensure, or workshop attendance.

    Conclusions:

    • Significant score variation and low evaluator agreement indicate unreliability in rating student performance with the current Michigan practical examination instrument.
    • The current assessment method does not consistently or reliably evaluate the technical competence of EMT candidates.
    • Improvements are needed to enhance the objectivity and consistency of EMT practical examinations.