Examining the validity of scores from an instrument designed to measure metacognition of problem solving

The authors' purpose in this study was to gather evidence on the validity of scores on an instrument designed to evaluate levels of metacognition in problem solving--Swanson's 15-item metacognition questionnaire (1990, 1993). Forty-three participants took part in this study: 21 professors and 22 graduate students from several Western and Midwestern campuses in the fields of psychology and educational psychology. A Kendall Coefficient of concordance (W) was performed to determine the consistency of ranked responses for professors, students, and both groups combined, and to determine whether their responses matched those of the scoring key for the instrument we studied. This analysis indicated problems for some of the responses to this instrument, as demonstrated through the ranking task developed in this study. Many of the professors disagreed with the ranking of the responses suggested by Swanson. There is certainly a need for more instruments to validly measure metacognition. This instrument is one that, with some modification, might be used more in research on metacognition.