Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test

Area of Science:

• Cognitive Psychology
• Psychometrics

Background:

• Standard interpretation of cognitive reflection tests (CRT) assumes correct responses are reflective and lured responses are unreflective.
• Prior process-tracing of mathematical CRTs has challenged this assumption.
• The verbal Cognitive Reflection Test (vCRT) is a new, validated, non-mathematical measure.

Purpose of the Study:

• To test the assumption that correct responses on the vCRT are reflective and lured responses are unreflective.
• To validate the vCRT as a measure of deliberate and conscious reflection.

Main Methods:

• Two studies (N = 201) utilized a validated think-aloud protocol.
• Data collected both in-person and online.
• Comparison with a control group to assess the impact of thinking aloud.

Main Results:

• Most correct responses on the vCRT involved reflection, and most lured responses lacked reflection.
• Thinking aloud did not significantly disrupt test performance.
• The vCRT generally satisfies the standard interpretation of reflection tests, with some exceptions.

Conclusions:

• The vCRT typically aligns with the standard interpretation of cognitive reflection tests.
• The vCRT appears to be a valid measure of the two-factor explication of reflection (deliberate and conscious).
• The think-aloud protocol is a suitable method for process-tracing the vCRT without disrupting performance.