Six misconceptions about large language models: A minimal model and diagnostic taxonomy

Area of Science:

• Artificial Intelligence
• Computational Linguistics
• Cognitive Science

Background:

• Large language models (LLMs) are increasingly integrated into scientific, educational, and governance systems.
• Current debates about LLMs are often shaped by simplistic 'folk theories,' such as 'just autocomplete' or 'emergent agents.'
• These intuitive models capture some aspects of LLMs but fail to represent their full complexity.

Purpose of the Study:

• To propose a minimal working model for understanding LLM-based systems.
• To diagnose and correct common misconceptions about LLM capabilities, mechanisms, and impacts.
• To provide a framework for evaluating AI capabilities, designing systems, and informing governance.

Main Methods:

• Introduced a minimal working model for LLM systems based on four key distinctions: pretraining vs. deployed systems, learned distribution vs. samples, types of memory (parametric, contextual, external), and task competence vs. agency.
• Analyzed six common misconceptions (next-token prediction, regression to the mean, training-data regurgitation, model memory, alignment, understanding) using the proposed model.
• Applied the framework to analyze publisher AI policies as case studies in AI governance.

Main Results:

• The minimal working model helps to clarify the distinctions between different aspects of LLM operation and capabilities.
• Diagnosed how specific misconceptions arise from conflating these distinctions.
• Demonstrated how AI policies can inadvertently perpetuate these misconceptions and offered methods for correction.

Conclusions:

• LLMs should be viewed as simulators of discourse and task performance, avoiding the simplistic 'parrot' or 'mind' dichotomy.
• The proposed model offers a diagnostic toolkit to identify and rectify errors stemming from folk theories about LLMs.
• Accurate understanding is crucial for effective LLM system design, capability evaluation, and responsible governance.