Asymptotic theory of in-context learning by linear attention

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Transformers demonstrate in-context learning (ICL), a key capability for task execution without explicit training.
• Unresolved questions persist regarding sample complexity, pretraining task diversity, and context length crucial for effective ICL.

Purpose of the Study:

• To precisely answer questions about ICL requirements using an exactly solvable model.
• To analyze the impact of pretraining task diversity and sample complexity on Transformer generalization.

Main Methods:

• Developed an exactly solvable model of ICL for a linear regression task using linear attention.
• Derived sharp asymptotics for the learning curve in a specific scaling regime (infinite token dimension, proportional context length and task diversity, quadratic pretraining examples).

Main Results:

• Demonstrated a double-descent learning curve as pretraining examples increase.
• Identified a phase transition between low and high task diversity regimes, distinguishing memorization from genuine ICL and generalization.

Conclusions:

• Theoretical insights into ICL mechanisms were derived and validated empirically.
• High task diversity is essential for Transformers to achieve genuine ICL and generalize beyond pretrained tasks.