AI-accelerated meta-analysis in psychology: Large language models code study properties with high accuracy.

Area of Science:

• Psychology
• Data Science
• Research Methodology

Background:

• Meta-analyses often test theoretical accounts by examining study characteristics.
• Manual coding of these characteristics is time-consuming and prone to bias.
• Current methods often involve double-coding only a subset of studies.

Purpose of the Study:

• To evaluate the capability of large language models (LLMs) in coding higher-level conceptual dimensions for psychological meta-analyses.
• To assess if LLMs can serve as reliable double-coders for ensuring data integrity.

Main Methods:

• Investigated OpenAI's GPT-5 and Google's Gemini 2.5 Pro for study coding.
• Tested LLM performance against manually coded data from three recent psychological meta-analyses.
• Provided limited information to LLMs to simulate real-world constraints.

Main Results:

• Both LLMs demonstrated high accuracy (over 92%) in replicating study codes.
• LLMs successfully coded higher-level conceptual dimensions, not just explicit information.
• Performance was robust despite limited input information.

Conclusions:

• LLMs show significant potential as effective double-coders in meta-analysis.
• LLM-assisted coding can enhance data integrity and efficiency in psychological research.
• This technology offers a practical solution to the laborious nature of manual data extraction.