Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations

Area of Science:

• Artificial Intelligence
• Neuroscience
• Cognitive Science

Background:

• Large language models (LLMs) sometimes report their strategies, suggesting metacognition.
• Metacognitive abilities in AI enhance capabilities but raise safety concerns regarding process transparency.
• Understanding LLM metacognitive limits is crucial for AI safety and oversight.

Purpose of the Study:

• To quantify the metacognitive ability of LLMs to explicitly report and control their internal activation patterns.
• To investigate the factors influencing LLM's capacity for monitoring neural activations.
• To assess the implications of LLM metacognition for AI safety.

Main Methods:

• Introduced a neuroscience-inspired neurofeedback paradigm.
• Trained LLMs using sentence-label pairs linked to specific neural activation directions.
• Quantified LLM performance in reporting and controlling internal activations.

Main Results:

• LLMs can learn to report and control specific activation patterns.
• Performance depends on training data, semantic interpretability of neural directions, and variance explained.
• Revealed a low-dimensional "metacognitive space" within LLMs, much smaller than their neural space.

Conclusions:

• LLMs possess a limited capacity to monitor their internal neural mechanisms.
• Quantifiable metacognitive abilities in LLMs have significant implications for AI safety and alignment.
• Further research is needed to explore the full extent and limitations of LLM metacognition.