Assessment of Advanced Life Support competence when combining different test methods--reliability and validity

Robust assessment of Advanced Life Support (ALS) competence is paramount to the credibility of ALS-provider certification and for estimating the learning outcome and retention of ALS competence following the courses. The European Resuscitation Council (ERC) provides two sets of MCQs and four Cardiac Arrest Simulation Test (CASTest) scenarios for the assessments according to guidelines 2005.

To analyse the reliability and validity of the individual sub-tests provided by ERC and to find a combination of MCQ and CASTest that provides a reliable and valid single effect measure of ALS competence.

Two groups of participants were included in this randomised, controlled experimental study: a group of newly graduated doctors, who had not taken the ALS course (N=17) and a group of students, who had passed the ALS course 9 months before the study (N=16). Reliability in terms of inter-rater agreement and generalisability across skills scenarios were estimated. Validity was studied in terms of equality of test difficulty and ability to discriminate performance between the groups.

Inter-rater agreement on checklist scores were generally high, Intraclass Correlation Coefficients between 0.766 and 0.977. Inter-rater agreements on pass/fail decisions were not perfect. The one MCQ test was significantly more difficult than the other. There were no significant differences between CASTests. Generalisability theory was use to identify a composite of MCQ and CASTest scenarios that possessed high reliability, equality of test sets, and ability to discriminate between the two groups of supposedly different ALS competence.

ERC sub-tests of ALS competence possess sufficient reliability and validity. A combined ALS score with equal weighting of one MCQ and one CASTest can be used as a single measurement of ALS competence.