Meta predictive learning model of languages in neural circuits.

Area of Science:

• Computational Neuroscience
• Artificial Intelligence
• Machine Learning

Background:

• Large language models (LLMs) demonstrate remarkable capabilities using self-attention mechanisms.
• The human brain's language processing may differ from LLM principles, sparking debate on brain computation and artificial self-supervision.
• The predictive coding framework is a prominent brain computation hypothesis, but its role in language processing and credit assignment is unclear.

Purpose of the Study:

• To propose and validate a mean-field learning model within the predictive coding framework.
• To investigate the connection between brain computation, next-token prediction, and general intelligence.
• To explore the role of predictive coding and credit assignment in language processing.

Main Methods:

• Developed a mean-field learning model based on the predictive coding framework.
• Assumed synaptic weights follow a spike and slab distribution, training only the distribution.
• Validated the meta-predictive learning model on handwritten digit classification (sequential pixel input) and language corpora (toy and real).

Main Results:

• The model achieved successful classification on sequential data, including handwritten digits and language corpora.
• Post-learning, most connections became deterministic, while output connections exhibited higher variability.
• Network ensemble performance scaled continuously with data load and improved with more training data, mirroring LLM emergent behavior.

Conclusions:

• The proposed meta-predictive learning model offers a framework for understanding brain computation in language processing.
• The findings suggest a potential link between predictive coding, next-token prediction in LLMs, and emergent general intelligence.
• The model provides a foundation for future research bridging neuroscience and artificial intelligence.