Related Experiment Video
Updated: May 11, 2026

Problem-Solving Before Instruction (PS-I): A Protocol for Assessment and Intervention in Students with Different Abilities
Published on: September 11, 2021
Comparative evaluation of multiple choice question formats. Introducing a knowledge score
Sheikh I Rahim1, Mahdi S Abumadini
1Department of Psychiatry, King Fahd Hospital of the University, College of Medicine, King Faisal University, PO Box 10401, Al-Khobar 31952, Kingdom of Saudi Arabia. Tel. +966 (3) 8957911. Fax. +966 (3) 8993996.
Objective:
Over the last 6 years different multiple choice question (MCQ) formats have been used in postgraduate examinations for trainees in psychiatry. In phase 1 - K-type True/False (T/F) items with negative marking; in phase 2 combined T/F and type-A one-best answer (OBA) questions without negative marking; in phase 3 exclusively OBA without negative marking. The study compares the gross scores (GS) obtained with different MCQ formats, and introduces knowledge score (KS).
Methods:
The study was conducted in the Saudi Council for Health Specialties, Riyadh, Kingdom of Saudi Arabia from 1996 to 2002. The mean percentile scores obtained by all postgraduate trainees sitting any Part I or Part II Saudi Board Examination in Psychiatry were subjected to a comparative analysis.
Results:
A total of 110 candidates sat 18 examinations returning 143 papers containing a total of 32,375 MCQ options. Phase 1 generated lowest overall mean GS (47.8%), phase 3 occupied an intermediate position (53.1%) and phase 2 produced the highest score (68.3%). The KS, to the contrary, generated strikingly similar results for all the 3 phases (47.8, 50.5 and 49.5%) indicating that the marked differences in the GS were probably related to benefits obtained from guessing in the absence of negative marking. In this respect, the OBA produced considerably higher KS scores than the T/F, presumably due to its facilitating extra benefits from cueing, partial knowledge and judgement.
Conclusion:
Different MCQ formats generate dissimilar quantitative results. The OBA format seems superior to the T/F format in crediting judgement and application of knowledge. In non-negatively marked MCQ tests, the suggested KS provides results comparable to those of negatively marked tests. Pass marks in MCQ tests should be calibrated according to the used format.
Related Concept Videos
Introduction to z Scores
z scores help...
Introduction to z Scores
z scores help...
Review and Preview
Percentiles are a type of fractile that partition data into...
Multiple Comparison Tests
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
z Scores and Unusual Values
This score indicates how far a value is from the mean in terms of standard deviation. For example, if a data value has a z score of +1, the researcher can infer that the particular data value is one standard deviation above the mean. If another data value...
Cochran's Q Test