Large language models for full-text methods assessment: a case study on mediation analysis

Area of Science:

• Psychiatry and Psychology Research
• Artificial Intelligence in Scientific Literature Review

Background:

• Systematic reviews are crucial for evidence synthesis but are labor-intensive, especially for extracting detailed methodological information from full-text articles.
• Assessing causal assumptions and methodological best practices in psychiatry and psychology studies requires expert-level review.

Purpose of the Study:

• To evaluate the performance of large language models (LLMs) in conducting full-text methodological reviews of mediation analysis studies.
• To compare LLM capabilities against human expert-level review on key causal assumptions and best practices.

Main Methods:

• Six LLMs (ChatGPT, Claude, Gemini) were tested on 180 full-text mediation analysis articles previously reviewed by methodologists.
• LLMs assessed 14 binary methodological criteria, with performance measured against expert consensus using accuracy, precision, recall, F1, AUC, and PR-AUC.

Main Results:

• LLM performance strongly correlated with human reviewers (accuracy correlation 0.71, F1 correlation 0.95).
• Advanced LLMs achieved near-human accuracy on explicit features but were up to 15% less accurate on inference-intensive tasks.
• Model accuracy decreased with longer documents, and common errors involved overinterpretation and misinterpretation of technical terms.

Conclusions:

• Findings support a criterion-specific human-AI collaboration strategy for full-text methodological assessment.
• A reproducible framework is provided for future LLM testing in evidence synthesis settings.