RAPID: Reliable and efficient Automatic generation of submission rePortIng checklists with large language moDels

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Scientific Publishing

Background:

• Accurate and comprehensive reporting in medical research is crucial for reproducibility and clinical decision-making.
• Existing manual checklist generation processes can be time-consuming and prone to errors.
• The integration of Artificial Intelligence (AI) offers potential solutions for enhancing reporting quality and efficiency.

Purpose of the Study:

• To evaluate the performance of RAPID, an automated reporting checklist generation tool.
• To assess the tool's effectiveness in utilizing large language models (LLMs) and retrieval augmentation generation (RAG) for medical reporting.

Main Methods:

• Developed a RAG architecture using LLMs to create the RAPID tool.
• Collected and manually annotated 91 published randomized controlled trials (RCTs) according to CONSORT and CONSORT-AI guidelines.
• Dataset included 50 RCTs without AI and 41 RCTs with AI interventions.

Main Results:

• RAPID incorporated all 37 CONSORT reporting items, achieving 92.11% average accuracy and 81.14% content consistency.
• For CONSORT-AI, RAPID included 11 AI-specific items, reaching 83.81% average accuracy and 72.51% content consistency.
• The tool demonstrated state-of-the-art performance on both datasets compared to other methods.

Conclusions:

• RAPID effectively automates checklist generation, saving time and improving efficiency for medical authors, researchers, editors, and reviewers.
• The tool exhibits strong scalability and adaptability to various medical reporting guidelines without extensive retraining.
• RAPID represents a significant advancement in AI-assisted medical reporting quality assurance.