A pilot experiment on the inter-examiner reliability of short essay questions

Although long essay questions have been shown generally to have a low inter-examiner reliability, they are still much used in undergraduate medical examinations, evidently because examiners feel that essays have greater inherent validity than some of the modern 'objective' techniques such as multiple choice questions (MCQs). Evidence exists in the literature to suggest that the reliability of short (c. 10 minutes) essay questions may be higher. A pilot experiment is reported in which students answered six questions requiring a short essay or extended notes (10 minutes per question): these 'SEQs' were each marked by two examiners. With the exception of one question, agreement between examiners was good. The implications of the experiment are considered to be: (a) that SEQs may well have a useful role in a broadly-based examination system; (b) there is a need for more experimentation with this type of question, which can arguably have greater validity than the (highly reliable) MCQ; and (c) that dual marking of all essay scripts is most desirable.