New machine-marked tests for selection into core medical training: evidence from two validation studies

Area of Science:

• Medical Education
• Psychometrics
• Healthcare Professional Training

Background:

• Selection into Core Medical Training (CMT) in the UK requires robust assessment methods.
• Machine-marked tests (MMTs), including clinical problem-solving and situational judgement tests, have demonstrated validity in General Practice (GP) training selection.
• The applicability of these MMTs to CMT selection requires further investigation.

Purpose of the Study:

• To evaluate the psychometric properties and predictive validity of MMTs for Core Medical Training (CMT) selection.
• To determine if MMTs can serve as a valid shortlisting tool for CMT applicants.
• To assess the incremental validity of MMTs over existing selection processes.

Main Methods:

• A longitudinal study design was employed, analyzing data from two independent applicant cohorts (2008 and 2009).
• Psychometric properties (reliability) of the MMTs were assessed in CMT samples.
• Correlations between MMT scores and subsequent CMT interview performance were calculated.

Main Results:

• Both MMTs demonstrated good reliability in CMT samples, comparable to GP training samples.
• MMTs were significant predictors of CMT interview performance, with strong correlations observed in both cohorts (r = 0.56 in 2008, r = 0.61 in 2009).
• MMTs provided incremental validity beyond the current shortlisting procedures.

Conclusions:

• The MMTs validated for GP training selection are appropriate and effective for Core Medical Training (CMT) shortlisting.
• These tests represent a significant advancement in medical selection methodology.
• Implementing MMTs can enhance the fairness and predictive accuracy of the CMT selection process.