Assessment of Advanced Life Support competence when combining different test methods--reliability and validity

Area of Science:

• Medical Education
• Emergency Medicine
• Cardiopulmonary Resuscitation

Background:

• Robust assessment of Advanced Life Support (ALS) competence is crucial for certification and evaluating learning outcomes.
• The European Resuscitation Council (ERC) offers multiple-choice questions (MCQs) and Cardiac Arrest Simulation Tests (CASTests) for ALS assessment.

Purpose of the Study:

• To analyze the reliability and validity of individual ERC sub-tests.
• To identify an optimal combination of MCQ and CASTest for a single, reliable measure of ALS competence.

Main Methods:

• A randomized controlled study compared newly graduated doctors (N=17) with students who had passed the ALS course 9 months prior (N=16).
• Reliability was assessed via inter-rater agreement and generalizability across scenarios.
• Validity was evaluated by comparing test difficulty and discrimination between groups.

Main Results:

• Inter-rater agreement on checklist scores was high (ICC 0.766–0.977), but pass/fail decisions varied.
• One MCQ was significantly more difficult than the other; CASTests showed no significant differences.
• A composite score using one MCQ and one CASTest demonstrated high reliability, test set equality, and discriminative ability.

Conclusions:

• ERC sub-tests for ALS competence demonstrate adequate reliability and validity.
• A combined ALS score, equally weighting one MCQ and one CASTest, serves as a valid single measure of ALS competence.