Spin-glass model of in-context learning.

Area of Science:

• Artificial Intelligence
• Statistical Physics
• Machine Learning

Background:

• Large language models (LLMs) demonstrate remarkable in-context learning (ICL) capabilities, predicting outcomes from prompts without explicit retraining.
• The underlying mechanisms and physical interpretations of ICL remain largely unexplained, posing a significant challenge to the field.

Purpose of the Study:

• To provide a mechanistic interpretation of in-context learning in transformers.
• To connect the phenomenon of in-context learning to established principles in statistical physics.

Main Methods:

• A simplified transformer model with linear attention was analyzed.
• This model was mapped to a spin glass model with real-valued spins.
• The spin glass model's parameters (couplings and fields) were used to explain data disorder and weight interactions during pretraining.

Main Results:

• The spin glass model elucidates how weight parameters interact during pretraining, explaining prediction without further training.
• Increased task diversity was shown to drive the emergence of in-context learning in single-instance learning scenarios.
• The Boltzmann distribution's convergence to a unique solution of weight parameters was linked to task diversity and ICL.

Conclusions:

• The proposed spin glass model offers a tractable theoretical framework for interpreting LLM behavior.
• The study provides insights into how pretraining and task diversity enable in-context learning.
• This work opens new avenues for understanding puzzling properties of large language models through a physics-based lens.