Using multiple response true-false multiple choice questions

The Royal Australasian College of Surgeons (RACS) Board of Examiners, having agreed to replace Type K with Type X multiple choice questions (MCQ), undertook a study to determine the most appropriate method of scoring the latter. Retrospective scoring of candidates' responses in a previous examination, using six reported scoring techniques for Type X items, showed that the Middlesex technique yielded a mean score closest to the mean score of the Type K items which were being replaced. However, as candidates' responses are affected by instructions, and the latter could not be controlled, a prospective study was carried out on three examinations, in each of which the nature of Part C varied among the three papers, while Parts A and B were similar. Paper 1 contained Type K items in Part C, Paper 2 contained Type X items with instructions appropriate for the Middlesex technique, and Paper 3 contained Type X items scored +1 if correct, -1 if incorrect and 0 if omitted. Items sampling the same content areas were randomly distributed among the three papers. The results indicated that the two Type X systems yielded significantly lower mean scores than Type K, probably due to combinations characteristic of the latter, enabling candidates with partial knowledge to obtain a maximum score. The difference in mean scores between the two Type X systems was not significant. A significantly higher mean percentage score in reused Type X (Middlesex system) items over that of the same items when they were previously used as Type K items was attributed to the combined effects of credit for partial knowledge, practice on reused items and differences in mean candidate ability across years. Pearson product-moment correlation coefficients between pairs of Part C mean percentage scores at each examination were positive and significant. The Board decided to replace Type K with Type X items, and the +1, 0, -1 system of scoring was adopted as it was considered more efficient than the Middlesex system, which did not give credit for identifying false statements as false.