Validity of multiple-choice examinations in surgery

The difficulty of creating new, unambiguous, pertinent multiple-choice questions of a level appropriate to medical students implies that examinations must be compiled from a limited number of items. Furthermore, it is impossible to keep used questions inaccessible to all subsequent students. This study was undertaken to determine if these realities are compatible with examinations that are both valid and reliable. A pool of 480 multiple-choice questions was distributed to 232 students during the surgical clerkship. At the conclusion of each quarter, a 120-item multiple-choice examination that consisted of entirely new questions was administered (group I). These 960 questions were then made available to the next group of 218 students; each subsequent examination consisted of 50% new questions and 50% questions repeated verbatim from the publicized pool (group II). With the available pool now increased to 1200, the next examination consisted of 20% new and 80% repeat questions (group III). Reliability (internal consistency) was measured by the Kuder-Richardson-21 formula. Validity was measured by correlation between the multiple-choice examination and the average score of evaluations of each student by two oral examinations and five faculty members. Despite the expected increase in mean examination score, there is loss of neither reliability nor validity by inclusion of even 80% of items repeated from a large pool of multiple-choice questions that have been distributed to the students. Hence, instead of adding irrelevant, trivial, or inappropriate items or trying in vain to hide old examinations from new students, simple preparation of examinations from a large pool of questions is recommended. To insure fairness to all students, this pool should be made public knowledge.