A reappraisal of the use of the phi-coefficient in multiple choice examinations

The phi-coefficient of an item in a multiple choice question (MCQ) examination is often used to determine whether that item is suitable for re-use in future examinations. In order to be of value in this regard, the coefficient must be shown to be an objective and consistent index of the discriminating power of an MCQ item. The behaviour of the phi-coefficient (phi) was investigated in two one-from-five MCQ and two true/false examinations. It is shown that the magnitude of the phi-coefficient for the items in any examination is a function not only of the discriminating power of the items, but also of the magnitude of the countermark for incorrect responses, of the proportion of 'don't know' responses in the examination, and of how the 'don't know' responses are handled in the analysis. It is further shown that the reproducibility of the phi, when calculated for any pair of randomly selected portions of a class of students, is very poor. We conclude that the error of estimation of the phi is of the same order of magnitude as the phi's normal operative range.