A law of next-token prediction in large language models

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Large language models (LLMs) are powerful but often function as black boxes.
• Understanding internal data processing in LLMs is crucial for reliable predictions.

Purpose of the Study:

• To introduce a quantitative law explaining contextualized token embedding learning in LLMs.
• To provide insights into the layer-wise contribution to prediction accuracy.

Main Methods:

• Analysis of intermediate layer representations in pretrained LLMs.
• Quantitative modeling of embedding learning for next-token prediction.

Main Results:

• A precise, quantitative law governing embedding learning was identified.
• Each layer, from bottom to top, contributes equally to prediction accuracy.
• This phenomenon is consistent across diverse LLM architectures and datasets.

Conclusions:

• The discovered law offers a universal perspective on LLM internal workings.
• Findings provide actionable insights for LLM development, scaling, pretraining, and interpretation.