Automating data extraction in meta-research: A multi-model benchmark in network psychometrics papers.

Area of Science:

• Meta-research
• Bibliometrics
• Scientific publishing

Background:

• Manual data extraction in meta-research is labor-intensive, slow, and prone to errors.
• Automating this process is crucial for improving efficiency and scalability in scientific research.

Purpose of the Study:

• To evaluate the accuracy and efficiency of current large language models (LLMs) for extracting metascientific variables from research papers.
• To compare the performance of five different LLMs on this data extraction task.

Main Methods:

• An automated API-based pipeline was developed for batch processing of 43 network psychometrics papers.
• Five LLMs (Claude 4.6 Opus, Claude 4.5 Sonnet, Claude 4.5 Haiku, GPT-5.2, and GPT-5 mini) were tested for data extraction accuracy.
• Extraction accuracy was assessed for both verbatim information and subjective inferences.

Main Results:

• LLM extraction accuracy ranged from 79.6% to 91.3% across the tested models.
• Higher accuracy was observed for explicit, verbatim data compared to variables requiring complex inference.
• Most models demonstrated an ability to indicate uncertainty in ambiguous cases.

Conclusions:

• LLMs offer a viable and efficient alternative to manual data extraction in meta-research, substantially reducing time and costs.
• The suitability of LLMs depends on the nature of the data being extracted, with simpler variables yielding better results.
• Best practices and common pitfalls for automated LLM data extraction are identified to guide future applications.