Beyond human gold standards: A multimodel framework for automated abstract classification and information extraction

Area of Science:

• Artificial Intelligence in Medicine
• Evidence Synthesis
• Meta-Research

Background:

• Meta-research and evidence synthesis are resource-intensive.
• Large language models (LLMs) show promise but have variable performance.
• Reliability issues in LLM-assisted evidence synthesis hinder adoption.

Purpose of the Study:

• To develop and evaluate an agreement-based framework using small, open-source LLMs for meta-research tasks.
• To assess the framework's performance against human gold standards and large LLMs.
• To determine if LLM agreement can enhance accuracy and reduce manual workload in systematic reviews.

Main Methods:

• Implemented an agreement-based framework with multiple small LLMs (<10B parameters).
• Tested on 1020 rheumatology randomized controlled trial abstracts for classification and patient number extraction.
• Developed an improved gold standard by re-examining abstracts with LLM-human disagreement.

Main Results:

• The agreement framework achieved accuracies above 95%, exceeding human gold standards on 85% of abstracts.
• Model combinations (e.g., 3/5, 4/6, 5/7 LLMs) demonstrated robust performance.
• Low-performing models contributed fewer accepted decisions, mitigating performance variability issues.

Conclusions:

• The agreement-based framework offers a scalable solution for abstract screening in systematic reviews.
• This approach can significantly reduce manual review burden while maintaining high accuracy and reproducibility.
• Human expertise can be reserved for complex cases, optimizing resource allocation in evidence synthesis.