Accuracy of rating scale interval values used in multiple mini-interviews: a mixed methods study.

Area of Science:

• Medical Education
• Psychometrics
• Assessment and Evaluation

Background:

• Multiple mini-interviews (MMI) use ordinal rating scales, assuming linear intervals between scores for candidate ranking.
• This assumption is often unvalidated, potentially introducing systemic bias and distorting final cumulative scores.
• Accurate interval values are crucial for the validity and fairness of MMI assessments.

Purpose of the Study:

• To establish validated rating scale values reflecting rater intent in the Canadian integrated French MMI (IFMMI).
• To quantitatively validate these values using an independent method.
• To explore the impact of revised scale values on final candidate rankings and understand rater rationale.

Main Methods:

• A 4-round consensus-group exercise with 42 experienced MMI interviewers to determine relative values for a 6-point scale (A-F).
• Parallel quantitative validation by comparing average scores assigned to candidates when specific scale options were used over three years.
• Simulation of the impact of new score values on final rankings using data from 43,412 IFMMI stations and 4,345 applicants.

Main Results:

• Experienced raters assigned values: B=86.7%, C=69.5%, D=51.2%, E=29.3% (vs. A=100%, F=0%).
• Quantitative validation yielded similar values: B=87.1%, C=70.4%, D=51.2%, E=31.8%.
• Qualitative analysis revealed raters perceive lower scores (serious offenses) as carrying more weight than higher scores (minor details).

Conclusions:

• Consulting experienced interviewers is effective for establishing MMI rating scale values that align with rater intent.
• Revised scale values, reflecting rater intent, improve the accuracy and fairness of the IFMMI assessment process.
• The study demonstrated potential shifts in candidate rankings (±21 to +5 percentiles) with the new scale, averaging ±1.4 percentiles.