Predicting undergraduate OSCE performance using traditional and construct-driven situational judgment tests at...

Area of Science:

• Medical Education
• Psychometrics
• Social Skills Assessment

Background:

• Situational Judgement Tests (SJTs) are widely used in undergraduate medical admissions to assess social skills and predict academic success.
• Two types of SJTs exist: traditional SJTs measuring general behavior and construct-driven SJTs targeting specific traits.
• No prior research has directly compared the predictive validity of these SJT types in medical school admissions.

Purpose of the Study:

• To investigate and compare the predictive validity of a traditional SJT (HAM-SJT) and a construct-driven SJT (CD-SJT) for objective structured clinical examination (OSCE) performance in medical students.
• To determine if the CD-SJT, with its agentic and communal scales, predicts OSCE outcomes in both low-stakes and high-stakes admission scenarios.

Main Methods:

• The study administered the HAM-SJT and the CD-SJT to two samples of medical students (low-stakes: n=159; high-stakes: n=160) during undergraduate admission.
• Student performance was evaluated using objective structured clinical examinations (OSCEs).
• Statistical analyses, including correlation and regression, were used to assess the predictive validity of the SJTs, controlling for factors like GPA, science test scores, and gender.

Main Results:

• A moderate positive correlation (r=0.20, p=0.009) was found between the communal scale of the CD-SJT and OSCE performance with trained patients in the high-stakes sample.
• The CD-SJT demonstrated incremental predictive value for OSCE performance beyond GPA, science test scores, and gender (ß = 0.18, p=0.020).
• The HAM-SJT showed a comparable raw correlation (r=0.14) which, after controlling for range restriction, reached r=0.20, similar to the CD-SJT's communal scale.

Conclusions:

• Construct-driven SJTs, particularly their communal aspects, show promise in predicting clinical performance in high-stakes medical admissions.
• The findings suggest that SJTs assessing specific constructs may offer valuable insights into a medical student's potential for effective patient interaction.
• This research provides initial evidence supporting the use of construct-driven SJTs in undergraduate medical admissions for enhanced candidate selection.