Improving Our Ability to Predict Resident Applicant Performance: Validity Evidence for a Situational Judgment Test

Area of Science:

• Medical Education
• Professionalism Assessment
• Physician Training

Background:

• Assessing professionalism and interpersonal skills is crucial for physicians but challenging in medical education.
• Situational judgment tests (SJTs) are a promising tool for measuring these noncognitive competencies.
• SJTs are used internationally for medical school and postgraduate selection but are under-investigated in US residency programs.

Purpose of the Study:

• To determine if an SJT measuring professionalism predicts resident performance on ACGME competencies and multisource professionalism assessments (MPAs).
• To evaluate the SJT's ability to predict both current and future resident performance.
• To assess the SJT's incremental validity over USMLE scores.

Main Methods:

• Developed an SJT assessing seven professionalism dimensions.
• Administered the SJT to residents across 21 programs at 2 institutions in 2017.
• Analyzed SJT and USMLE score validity in predicting ACGME competencies and MPAs over time (June 2017, Sept 2017, June 2018).

Main Results:

• SJT scores significantly predicted overall ACGME milestone performance (r = .13–.17) and MPA performance (r = .19–.21) at both assessment periods.
• The SJT also predicted specific ACGME competencies including patient care, systems-based practice, learning and improvement, communication skills, and professionalism.
• SJT scores demonstrated incremental validity over USMLE scores for predicting ACGME performance and MPA performance.

Conclusions:

• SJTs show potential for assessing noncognitive attributes in residency applicants.
• The SJT's incremental validity highlights the need for holistic candidate reviews incorporating both cognitive and noncognitive measures.
• SJTs can enhance the selection process for residency programs by providing a more comprehensive assessment of candidates.