AI-driven abstract generating: evaluating LLMs with a tailored prompt under the PRISMA-A framework

Area of Science:

• Artificial Intelligence in Scientific Publishing
• Natural Language Processing for Medical Literature

Background:

• Systematic reviews and meta-analyses are crucial for evidence-based dentistry.
• Reporting these reviews according to standards like the PRISMA Abstract Checklist can be time-consuming.
• Large language models (LLMs) offer potential solutions for efficient abstract generation.

Purpose of the Study:

• To compare the performance of ChatGPT-4o and Gemini Pro in generating structured abstracts from full-text systematic reviews.
• To evaluate abstract generation based on adherence to the PRISMA Abstract (PRISMA-A) Checklist.
• To assess the utility of a customized prompt for LLM-based abstract creation in orthodontics.

Main Methods:

• 162 systematic reviews and meta-analyses from Q1-ranked orthodontic journals (since 2019) were analyzed.
• Full-text articles were processed by ChatGPT-4o and Gemini Pro using a PRISMA-A Checklist-aligned prompt.
• Abstract quality was scored using a tailored Overall Quality Score (OQS) derived from the PRISMA-A checklist; reliability and model comparisons were statistically assessed.

Main Results:

• Both LLMs generated satisfactory PRISMA-A compliant abstracts.
• ChatGPT-4o consistently achieved higher Overall Quality Scores (OQS) than Gemini Pro (mean OQS 21.67 vs. 21.00, p < 0.001).
• ChatGPT-4o demonstrated superior performance in the 'Included Studies' and 'Synthesis of Results' sections, producing more complete and coherent outputs.

Conclusions:

• Both ChatGPT-4o and Gemini Pro can generate PRISMA-A compliant abstracts.
• ChatGPT-4o demonstrated superior quality and adherence to the PRISMA-A checklist compared to Gemini Pro.
• The developed prompt and LLM approach show promise for broader application in evidence-based dental and medical research to streamline reporting.