Development of a new critical thinking test using item response theory

Area of Science:

• Psychometrics
• Educational Assessment

Background:

• The Watson-Glaser Critical Thinking Appraisal (WGCTA) is a widely used measure of critical thinking.
• A limitation of the WGCTA is its high proportion of 2-alternative multiple-choice items, which may facilitate successful guessing.

Purpose of the Study:

• To introduce the Wagner Assessment Test (WAT), a novel instrument designed to assess critical thinking.
• To evaluate the psychometric properties of the WAT, particularly its ability to reduce successful guessing compared to the WGCTA.
• To examine the effectiveness of the WAT using item response theory (IRT) modeling.

Main Methods:

• Development of the WAT based on a 5-faceted view of critical thinking.
• Comparison of the WAT with the WGCTA, focusing on the number of response alternatives.
• Application of the 3-parameter logistic item response theory (IRT) model to data from 407 undergraduates.
• Assessment of convergent validity between full-pool IRT score estimates.

Main Results:

• The WAT, with its increased number of response alternatives, demonstrated more desirable test information and standard-error functions than the WGCTA.
• These improvements were observed even when the WAT item pool was shortened to match the WGCTA's length.
• Convergent validity between full-pool IRT score estimates for the WAT was strong (r = .69).

Conclusions:

• The Wagner Assessment Test (WAT) is a promising instrument for assessing critical thinking with enhanced psychometric properties.
• The design of the WAT effectively mitigates the issue of successful guessing prevalent in traditional critical thinking assessments.
• Further research is warranted to explore the implications of IRT-based measurement for critical thinking assessment.