Two models of raters in a structured oral examination: does it make a difference?

Area of Science:

• Medical Education
• Assessment and Evaluation

Background:

• Standardization of oral examinations has increased.
• Traditionally, a limited number of raters were employed.
• Prior research indicates that increasing rater numbers enhances reliability.

Purpose of the Study:

• To compare the reliability and scoring of two rater models in a multi-station structured oral examination.
• To investigate the presence of a halo effect in different rater models.

Main Methods:

• A multi-station structured oral examination was conducted.
• Two rater models were compared: station-specific raters and candidate-specific raters.
• Internal medicine residents' performance was evaluated simultaneously by two station-specific and two candidate-specific raters at each station.

Main Results:

• No significant differences were observed in overall examination scores between the two rater models.
• Candidate-specific raters demonstrated higher reliability.
• Evidence of a halo effect was suggested for candidate-specific raters, potentially influencing individual station outcomes.

Conclusions:

• The candidate-specific rater model offers greater overall reliability in structured oral examinations compared to the station-specific model.
• The potential halo effect associated with candidate-specific raters warrants consideration as it may impact the assessment of individual performance.
• Further research is needed to fully understand and mitigate the halo effect in rater models.