A practice oral examination rating scale--inter-observer reliability

Area of Science:

• Medical Education
• Professional Assessment

Background:

• Certification examinations are critical for validating specialist competence.
• Ensuring reliability and consistency in rater scoring is a persistent challenge in medical education.

Purpose of the Study:

• To evaluate the inter-rater reliability of scores in simulated clinical oral examinations.
• To identify potential sources of variability and disagreement among raters in certification assessments.

Main Methods:

• Three videotaped practice clinical oral examinations were assessed.
• A total of 29 certified specialists, involved in resident training, served as raters.
• Raters evaluated the examinations, and scores were analyzed for consistency.

Main Results:

• Considerable differences in rating scores were observed in two of the simulated examinations.
• These score discrepancies led to disagreements on pass/fail status for candidates.
• Potential examiner "contamination" or bias was noted as a factor influencing ratings.

Conclusions:

• Current rating practices for certification examinations exhibit significant variability.
• The development of more reliable and standardized assessment instruments is urgently needed.
• Addressing rater variability is essential as professional college policies for examinations evolve.