Measurement precision for repeat examinees on a standardized patient examination

Area of Science:

• Medical Education
• Psychometrics
• Clinical Skills Assessment

Background:

• Examinees repeating SP-based clinical skills exams often show significant score improvements.
• This raises questions about the reliability of measurements on initial or repeated attempts.
• The Step 2 Clinical Skills Examination (CS) is a critical component of medical licensing.

Purpose of the Study:

• To evaluate the score precision for examinees repeating the SP-based Step 2 CS exam.
• To determine if large score gains on repeat attempts are due to measurement error.
• To analyze measurement precision across different examinee groups.

Main Methods:

• Generalizability theory was employed to compute conditional standard errors of measurement (SEM).
• Data included approximately 60,000 single-take and 5,000 repeat examinees from 2007-2009.
• The analysis focused on communication and interpersonal skills ratings.

Main Results:

• Conditional SEMs for single-take and repeat examinees were nearly identical across the score scale.
• Measurement precision was consistent for US graduates and International Medical Graduates (IMGs).
• Examinees with large score changes did not exhibit poorer measurement on either attempt.

Conclusions:

• Large score increases in repeat examinees on the SP-based Step 2 CS exam are unlikely due to significant measurement errors.
• The assessment demonstrates consistent precision for examinees, regardless of repeat attempts or background.
• Findings support the reliability of the Step 2 CS exam in measuring clinical skills.