Measuring human metacognition in GPT-assisted cognition: development and psychometric validation of a novel scale

The current age witnessed the development of artificial intelligence and how generative AI tools altered engagement of individuals in their cognitive tasks, which raises important questions about the role of human metacognition in AI-assisted environments. While traditional metacognition theories have extensively examined how people monitor, regulate and evaluate their cognitive processes, it is worth noting that it was for a human only learning context and do not adequately capture metacognitive awareness when cognition is partially shared with AI. The absence of context-specific measurement instruments limits the ability of researchers to systematically investigate how individuals consciously engage with AI during various tasks. Addressing this gap, the present study develops and validates a psychometric instrument designed to measure GPT-assisted metacognitive awareness.

Using a multi-stage scale development procedure, items were generated based on classical metacognitive theory and adapted to AI-mediated cognitive contexts. The scale was evaluated through expert content validation, exploratory factor analysis (EFA), confirmatory factor analysis (CFA) and reliability and construct validity assessment using separate samples.

EFA initially identified a three-factor structure consisting of 23 items representing distinct but related dimensions of GPT-assisted metacognitive awareness. However, subsequent CFA and validity assessments indicated substantial overlap between two factors, leading to a refined two-factor model demonstrating superior construct validity and parsimony. The overall scale demonstrated strong internal consistency.

The findings suggest that metacognitive awareness in GPT-assisted contexts represents a measurable cognitive construct that extends beyond traditional models of human only metacognition. The proposed scale provides a foundational tool for examining conscious versus passive AI use, understanding cognitive regulation in AI-assisted learning and informing educational and policy discussions surrounding responsible AI integration.

By offering a validated measurement framework, the study contributes to emerging research on human-AI cognitive interaction and metacognitive regulation in the age of generative AI and in turn support SDG Goal 3, 4, and 9.