[Use of model answers in evaluating reliability of written tests]

The use of model answers (memoranda) for the evaluation at tertiary level of written test or examination papers generally is not considered important. Model answers, however, set standardized criteria according to which they can be evaluated. The purpose of this study was to determine the effect of the use of a model answer on rater reliability during the evaluation of written test papers in Community Dentistry. Fifteen official written test papers were randomly selected. The papers were photocopied and handed to four evaluators namely, the author of the question paper and model answer, an educational expert (calibrated non-dentist) who was given a model answer and two subject experts (specialist and registrar) of whom one was given a model answer and was calibrated. The marks of each evaluator were correlated with those of the author of the question paper by means of Spearman's reliability co-efficient. Based on the marks allocated, interrater reliability between the lecturer and calibrated evaluators using a model answer ranged between 0.65 and 0.86 and the evaluations done without a memorandum or calibration gave a r-value of 0.59 (p less than 0.02). Although the quality of the memorandum is an important consideration, this study shows that the use of a model answer and calibration results in a distinct improvement in rater reliability.