Ontology-constrained multi-LLM scoring of hypothesis support in the predictive processing literature

Area of Science:

• Neuroscience
• Computational Neuroscience
• Artificial Intelligence

Background:

• Interdisciplinary fields like predictive coding neuroscience face fragmentation due to diverse methods and theories.
• Conventional meta-analysis struggles to synthesize this complex literature spanning computational theory, electrophysiology, imaging, behavior, and modeling.
• A structured approach is needed to map heterogeneous evidence and identify areas of consensus and disagreement.

Purpose of the Study:

• To develop and evaluate a local multi-large language model (LLM) pipeline for ontology-constrained literature synthesis in predictive coding neuroscience.
• To create a quantitative framework for analyzing agreement and disagreement across studies within a complex scientific domain.
• To map the hypothesis space of predictive coding and identify patterns of structured disagreement.

Main Methods:

• A local multi-LLM pipeline was developed to read papers, extract evidence, incorporate figure descriptions, and validate outputs against an expert glossary.
• A predictive-coding glossary of 36 concepts grouped into three hypotheses was manually defined.
• Ten local LLMs scored 31 studies on agreement/disagreement with glossary factors in local and global oddball contexts, enabling pairwise analysis and hypothesis-space mapping.
• Hypothesis-space temperature was defined as a metric for geometric dispersion of studies.

Main Results:

• The LLM council enabled pairwise study-agreement analysis, cross-model comparison, and 3D hypothesis-space mapping.
• Agreement varied across hypotheses, revealing structured disagreement, particularly between local and global oddball paradigms.
• Hypothesis-space temperature was lower for local oddball contexts and higher for global oddball contexts, indicating greater dispersion.
• Vectors of change between experimental contexts were estimated from the scoring geometry.

Conclusions:

• Local multi-LLM councils can generate auditable disagreement measurements to map heterogeneous literatures into quantitative evidence spaces.
• This framework provides a novel method for cross-study hypothesis mapping where traditional meta-analysis is insufficient.
• The approach demonstrates potential for synthesizing complex scientific domains and understanding structured disagreement within them.