Improvement of reliability of an oral examination by a structured evaluation instrument

The main purposes of this study were to estimate the reliability of oral examinations administered to medical students during a clinical clerkship and to improve the reliability of this evaluation technique. In the first part of the study, the reliability of oral examinations as traditionally administered was estimated. The average intraclass reliability coefficient for these examinations was .48. Cassette recordings of these oral examinations were also rated by the faculty members. The average intraclass reliability coefficient of the ratings of the taped performances was .82. In the second part of the study, the reliability of oral examinations was investigated with the raters using a newly developed evaluation form. The average intraclass reliability of the oral examination using the evaluation form was .67, a noticeable increase over the .48 obtained without the form. The average intraclass reliability of ratings made from tape recordings of these oral examinations was .62.