Large language models surpass human experts in predicting neuroscience results.

Area of Science:

• Neuroscience
• Artificial Intelligence
• Scientific Discovery

Background:

• Synthesizing vast amounts of scientific research is crucial for discovery but challenges human cognitive limits.
• Large language models (LLMs) offer a potential solution for integrating complex research findings.

Purpose of the Study:

• To evaluate the capability of LLMs in predicting experimental outcomes in neuroscience.
• To introduce BrainBench, a novel benchmark for assessing predictive performance in neuroscience research.

Main Methods:

• Development of BrainBench, a benchmark dataset designed for forward-looking prediction of neuroscience results.
• Evaluation of general LLMs and a specialized LLM (BrainGPT) trained on neuroscience literature.
• Comparison of LLM predictive performance against human expert performance.

Main Results:

• LLMs demonstrated superior performance in predicting experimental outcomes compared to human experts.
• BrainGPT achieved even higher accuracy, outperforming both general LLMs and human experts.
• LLM prediction accuracy correlated with confidence levels, mirroring expert performance patterns.

Conclusions:

• LLMs, particularly specialized ones like BrainGPT, show significant promise in aiding scientific discovery by predicting research outcomes.
• The confidence-accuracy correlation suggests LLMs can provide reliable insights, potentially augmenting human research capabilities.
• The methodology is transferable to other knowledge-intensive scientific fields beyond neuroscience.