The validity of selection assessments for medical specialty selection: examining prediction across different training...

The Multi-Specialty Recruitment Assessment (MSRA) is a proctored assessment, comprising a clinical problem-solving (CPS) test and a situational judgement test (SJT), often combined with selection interviews, for admission to multiple postgraduate medical specialty training programmes in the UK. The study examined the predictive validity of these selection assessments (i.e. the extent to which they effectively predicted later performance outcomes).

Data were obtained from the UKMED dataset and analyzed for ten UK medical specialties (~13 000 unique applications) to explore relationships between selection data (CPS, SJT, and selection interview) and subsequent performance outcomes; Annual Review of Competence Progression (ARCP) and national professional examination scores, comprising Applied Knowledge Tests (AKTs) and Objective Structured Clinical Examinations (OSCEs).

Results supported the predictive validity of the MSRA and interview scores with some evident trends across different criteria. For example, the CPS tended to be most predictive of the AKT performance, and the SJT (which assesses non-academic skills) tended to be more strongly predictive of OSCE than AKT performance. The predictive validity of the selection interviews beyond the MSRA varied by specialty.

Results provide evidence of the predictive validity for both the MSRA and interviews across a range of specialties, providing insights into how best to design and combine different selection methods to balance key considerations such as reliability, validity, efficiency, and positive candidate feedback. The findings of this research also have implications for other healthcare selection settings. Key messages • What is already known on this topic - Previous research has already demonstrated that the MSRA shows predictive validity in the context of General Practice and Psychiatry selection, and has begun to explore the validity of the MSRA and interviews in further specialties. However, there is a lack of recent research exploring the predictive validity of MSRA in selection processes across other specialties, using a range of criterion measures and longitudinal performance outcomes, such as exam data. • What this study adds - The MSRA demonstrates validity with Annual Review of Competence Progression (ARCP) and national professional examination scores in postgraduate medical specialty training, across a range of different medical specialties. • How this study might affect research, practice, or policy - The results of this study can be used to inform future policy and decision making regarding the use of the MSRA alongside selection interviews and to enhance the reliability, validity, efficiency, and candidate experience for selection in future. Further, results provide valuable insights about the validity of such methods which may be informative to other settings internationally.