Measuring higher cognitive levels by multiple choice questions: a myth?

Available evidence is inconclusive as to the ability of multiple choice items to measure different taxonomic levels of the cognitive domain. The present study analysed the tests of the Examen de Synthèse for the years 1982, 1983 and 1984. Items used in the study were those for which a consensus was reached between three judges and committees for a given taxonomic level. The initial part of the study showed that judges do not classify items at random but according to a mental representation which is individual, personal and relatively stable. In examining results obtained by students, the study failed to show any significant difference in item difficulty or discrimination for items classified as measuring memorization, interpretation of data and problem-solving. Correlations between results (scores and ranks) obtained for items involving memorization and those obtained for items involving higher cognitive levels fail to show that different traits are measured. If further studies corroborate these results, then future efforts should be directed at developing other instruments to measure higher cognitive levels.