The validity of a behavioural multiple-mini-interview within an assessment centre for selection into specialty...

Area of Science:

• Medical Education
• Psychometrics
• Healthcare Professional Training

Background:

• Specialty training selection in general practice utilized a National Assessment Centre (NAC) approach.
• This NAC combined a behavioral Multiple-Mini-Interview (MMI) with a written Situational Judgement Test (SJT).
• The study aimed to evaluate the reliability and validity of the MMI for assessing non-cognitive characteristics.

Purpose of the Study:

• To determine the reliability and validity of the Multiple-Mini-Interview (MMI) for selecting candidates into general practice specialty training.
• To explore the concurrent validity of the MMI in relation to the Situational Judgement Test (SJT).
• To identify sources of measurement error within the MMI process.

Main Methods:

• Variance components analysis was employed to estimate reliability and identify sources of measurement error.
• Further modeling was conducted to determine optimal MMI configurations for future assessments.
• The correlation between MMI scores and SJT scores was calculated to assess concurrent validity.

Main Results:

• Interviewer subjectivity accounted for 40% of score variance, while candidate variation explained 28%.
• A six-question MMI achieved a generalizability coefficient of 0.7; ten questions are needed for a coefficient of 0.8.
• Evidence for construct and concurrent validity was demonstrated through correlations with the SJT (r = 0.35) and its clinical knowledge subdomain (r = 0.25).

Conclusions:

• The MMI is a moderately reliable assessment tool within a National Assessment Centre framework.
• Interviewer subjectivity is a significant source of error, highlighting the need for enhanced interviewer training.
• MMIs require sufficient length for precise candidate ranking, and further theoretical work is needed to optimize the combination of written and performance-based assessments.