Interreader agreement on comprehensive essay examinations

Examination of faculty interreader reliability on essay-type comprehensive examinations revealed an overall agreement rate of 85.6%. No significant differences in agreement were found between departments. Examination of reader characteristics for those exams with 100 points variation between readers revealed no consistent differences. The findings support the use of one format with a structured scoring protocol for evaluating diverse clinical specialties.