Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Video

Updated: Jun 29, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

Large language models for automated PRISMA 2020 adherence checking.

Yuki Kataoka1, Ryuhei So2, Masahiro Banno3

  • 1Center for Postgraduate Clinical Training and Career Development, Nagoya University Hospital, Nagoya, Aichi, Japan; Center for Medical Education, Graduate School of Medicine, Nagoya University, Nagoya, Aichi, Japan; Scientific Research Works Peer Support Group (SRWS-PSG), Osaka, Japan; Department of Internal Medicine, Kyoto Min-iren Asukai Hospital, Kyoto, Japan; Department of Healthcare Epidemiology, Kyoto University Graduate School of Medicine / School of Public Health, Kyoto, Japan; Department of International and Community Oral Health, Tohoku University Graduate School of Dentistry, Sendai, Miyagi, Japan.

International Journal of Medical Informatics
|June 27, 2026
PubMed

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

A Scoping Review of Nonverbal Synchrony Between Individuals With Social Anxiety and Others.

Psychological reports·2026
Same author

Self-reported harassment during Japanese surgical residency: secondary analysis of a nationwide survey of newly board-certified surgeons.

Surgery today·2026
Same author

Methodological concerns in evaluating water exchange combined with distal attachments for adenoma detection.

Annals of gastroenterology·2026
Same author

Longer hours, limited returns? a post-hoc National survey analysis of overtime hours during surgical training in Japan.

Surgery today·2026
Same author

Effectiveness of a Fully Online Scientific Research Works Peer Support Group Model for Research Capacity Building Through Conducting Systematic Reviews Among Health Care Professionals: Retrospective Cohort Studies.

JMIR medical education·2025
Same author

Correction: A nationwide questionnaire survey to investigate facility‑based disparities in satisfaction and working conditions of surgical trainees in Japan: university hospitals, community hospitals, and hybrid‑type facilities.

Surgery today·2025
Summary

Structured checklists significantly enhance large language model (LLM) performance in evaluating PRISMA 2020 guideline adherence for systematic reviews (SRs). Human oversight remains crucial due to potential false positives.

Area of Science:

  • Medical Informatics
  • Artificial Intelligence in Research

Background:

  • Evaluating adherence to the PRISMA 2020 guideline is time-consuming in peer review.
  • A lack of standardized benchmarks hinders assessment of large language model (LLM) capabilities for this task.

Purpose of the Study:

  • To develop and evaluate a copyright-aware benchmark for assessing LLM performance in PRISMA 2020 guideline adherence.
  • To compare the effectiveness of different input formats for LLMs in this evaluation.

Main Methods:

  • A benchmark of 108 Creative Commons-licensed systematic reviews (SRs) was created.
  • Five checklist input formats (Markdown, JSON, XML, plain text, manuscript-only) were compared using ten LLMs on the Suda dataset.
  • The Markdown pipeline was validated with nineteen LLMs on the Tsuge dataset.

Related Experiment Videos

Last Updated: Jun 29, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

Main Results:

  • Structured PRISMA 2020 checklists improved accuracy to 78.7-79.7% compared to 45.2% for manuscript-only input.
  • Structured formats outperformed manuscript-only input, with no significant differences among structured formats.
  • Validation accuracy ranged from 68.5% to 86.0%, with notable sensitivity-specificity trade-offs; Qwen3-Max achieved 95.1% sensitivity and 49.3% specificity.

Conclusions:

  • Providing structured checklists substantially enhances LLM-based PRISMA assessment.
  • Human expert verification is still essential due to observed false positive rates before making editorial decisions.