Increasing alignment of large language models with language processing in the human brain

Area of Science:

• Cognitive Science
• Neuroscience
• Artificial Intelligence

Background:

• Transformer-based large language models (LLMs) offer insights into brain's meaning representation.
• Concerns exist regarding the validity of large LLMs due to extensive data and long context access.
• Instruction tuning is a key LLM technique beyond simple scaling.

Purpose of the Study:

• Investigate if instruction tuning enhances LLMs' ability to capture human brain's linguistic information.
• Compare base and instruction-tuned LLMs of varying sizes against human reading data.
• Assess the cognitive plausibility of LLMs for naturalistic language comprehension.

Main Methods:

• Compared base and instruction-tuned LLMs (varying sizes) with human data.
• Utilized eye-tracking and functional magnetic resonance imaging (fMRI) for brain activity.
• Analyzed data during naturalistic reading tasks.

Main Results:

• Increasing LLM size, rather than instruction tuning, resulted in a closer match to human brain activity.
• Base LLMs showed better alignment with human cognitive processes than instruction-tuned models.
• Model size is a more critical factor than instruction tuning for brain representational similarity.

Conclusions:

• Scaling LLMs is more effective than instruction tuning for mimicking human brain's linguistic processing.
• Findings challenge the cognitive plausibility of instruction-tuned LLMs for studying language comprehension.
• LLM size is a crucial consideration for their application in cognitive neuroscience research.