Development and validity evidence of a multiple-choice electrocardiogram test for undergraduate clerkship medical...

Accurate ECG interpretation is an essential skill required for medical school graduation. Multiple choice question (MCQ) tests are widely used to assess student competence due to the feasibility of administration; however, many existing MCQs ECG tests lack strong validity evidence. Our study aimed to develop validity evidence for an MCQ based ECG assessment using Kane's validity framework.

We analyzed the internal structure of a 22-item MCQ test. A modified Hofstee and Angoff method were used to set the passing standard. We compared MCQ scores with results from open-ended ECG quizzes to examine the relationship to a more "real-world" assessment. Pearson correlation coefficient and linear regression assessed the strength of the association between the two assessments and Welch's t-test compared open-ended scores between high and low MCQ performers.

Eighteen of the 22 items met performance criteria. A passing score of 11/18 was set to reflect the desired failure rate of 0-10%. The Pearson correlation coefficient between MCQ and open-ended ECG scores was 0.414, indicating a moderate positive linear relationship. Linear regression showed an R² of 0.19 between the two assessments. Students scoring in the lowest MCQ quartile had significantly lower open-ended scores (p < 0.001).

These findings provide validity evidence for an MCQ assessment that identifies students below a minimum competency threshold. While the test effectively identifies struggling learners, it lacks the sensitivity to differentiate proficiency levels above that threshold. Future work should explore efficient methods for assessing and providing feedback on open-ended ECG interpretation.