Related Experiment Video
Updated: Jun 29, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Large language models for automated PRISMA 2020 adherence checking
Yuki Kataoka1, Ryuhei So2, Masahiro Banno3
1Center for Postgraduate Clinical Training and Career Development, Nagoya University Hospital, Nagoya, Aichi, Japan; Center for Medical Education, Graduate School of Medicine, Nagoya University, Nagoya, Aichi, Japan; Scientific Research Works Peer Support Group (SRWS-PSG), Osaka, Japan; Department of Internal Medicine, Kyoto Min-iren Asukai Hospital, Kyoto, Japan; Department of Healthcare Epidemiology, Kyoto University Graduate School of Medicine / School of Public Health, Kyoto, Japan; Department of International and Community Oral Health, Tohoku University Graduate School of Dentistry, Sendai, Miyagi, Japan.
Structured checklists significantly enhance large language model (LLM) performance in evaluating PRISMA 2020 guideline adherence for systematic reviews (SRs). Human oversight remains crucial due to potential false positives.
Area of Science:
- Medical Informatics
- Artificial Intelligence in Research
Background:
- Evaluating adherence to the PRISMA 2020 guideline is time-consuming in peer review.
- A lack of standardized benchmarks hinders assessment of large language model (LLM) capabilities for this task.
Purpose of the Study:
- To develop and evaluate a copyright-aware benchmark for assessing LLM performance in PRISMA 2020 guideline adherence.
- To compare the effectiveness of different input formats for LLMs in this evaluation.
Main Methods:
- A benchmark of 108 Creative Commons-licensed systematic reviews (SRs) was created.
- Five checklist input formats (Markdown, JSON, XML, plain text, manuscript-only) were compared using ten LLMs on the Suda dataset.
- The Markdown pipeline was validated with nineteen LLMs on the Tsuge dataset.
Main Results:
- Structured PRISMA 2020 checklists improved accuracy to 78.7-79.7% compared to 45.2% for manuscript-only input.
- Structured formats outperformed manuscript-only input, with no significant differences among structured formats.
- Validation accuracy ranged from 68.5% to 86.0%, with notable sensitivity-specificity trade-offs; Qwen3-Max achieved 95.1% sensitivity and 49.3% specificity.
Conclusions:
- Providing structured checklists substantially enhances LLM-based PRISMA assessment.
- Human expert verification is still essential due to observed false positive rates before making editorial decisions.