Investigating the Performance of Second Language Medical Students on Lengthy Clinical Vignettes

Area of Science:

• Medical Education
• Psychometrics
• Linguistic Assessment

Background:

• The U.S. Medical Licensing Examination (USMLE) Step 2 assesses medical knowledge and clinical skills.
• Standardized testing often involves varying item lengths, potentially impacting examinees differently based on language proficiency.
• Understanding the influence of reading load on test performance is crucial for ensuring equitable assessment.

Purpose of the Study:

• To evaluate the impact of long clinical vignettes on medical students with limited English-language experience.
• To determine if English as a Second Language (ESL) status and English proficiency scores moderate the effect of item length on USMLE Step 2 performance.

Main Methods:

• Analysis of 25,012 U.S. Medical Licensing Examination® Step 2 examinee scores.
• Categorization of test items into five levels based on word count.
• Evaluation of examinee scores as a function of English-language experience (ESL status, English proficiency test scores).

Main Results:

• Longer test items were associated with lower scores across all examinee groups.
• Examinees with greater English proficiency consistently achieved higher scores.
• A statistically significant interaction between item word count and English-language experience was observed, but the effect size was very small (η² = .0004).

Conclusions:

• While longer test items present a slightly greater challenge for medical students with less English experience, the differential impact is minimal.
• The findings suggest that current USMLE Step 2 assessment is largely equitable across different levels of English proficiency.
• Continued monitoring of this small differential effect is recommended to ensure ongoing fairness in medical licensing examinations.