Reliability of different grading systems used in evaluating surgical students

Area of Science:

• Medical Education
• Assessment and Evaluation

Background:

• Accurate assessment of student performance is crucial in medical education.
• Various grading systems exist, but their reliability in reflecting rater consensus is often debated.
• Standardizing grading is essential for fair evaluation and program improvement.

Purpose of the Study:

• To compare the inter-rater agreement of five distinct grading systems in a surgery clerkship.
• To determine which grading system offers the highest reliability and best discrimination for student performance evaluation.
• To identify the optimal grading system for consistent and accurate assessment by faculty raters.

Main Methods:

• Conducted a reliability study involving 21 faculty raters assessing 16 students in a surgery clerkship.
• Utilized five grading systems: pass-fail, pass-fail-honors, letter grade, 1-10 numerical scale, and 1-100 numerical scale.
• Employed inter-rater agreement coefficients to evaluate both relative and absolute reliabilities.

Main Results:

• The letter grade and 1-10 numerical scale systems showed good discrimination and moderate to high reliability.
• Both systems required only five raters to achieve a reliability of 0.80.
• A majority of raters agreed on grades for 14/16 students using the letter grade system, versus only 4/16 for the 1-10 scale.

Conclusions:

• The letter grading system is favored for its superior inter-rater agreement and consistency in assessing student performance.
• Letter grades provide a more reliable basis for evaluation compared to numerical scales or pass-fail systems.
• This study supports the adoption of letter grading systems for enhanced reliability in medical education assessments.