MEDAI-LLM-SUMM: a reporting checklist for medical text summarization studies using large language models

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Medical Informatics

Background:

• Large Language Models (LLMs) show promise in medical text summarization, nearing human expert performance.
• Significant gaps exist in safety validation, evaluation frameworks, and clinical readiness for LLM applications.
• Current reporting guidelines inadequately address the nuances of medical text summarization research, with high hallucination rates observed (1.47%-61.6%).

Purpose of the Study:

• To develop MEDAI-LLM-SUMM, the first specialized reporting checklist for research on medical text summarization utilizing LLMs.
• To address critical deficiencies in existing reporting standards for medical LLM summarization studies.

Main Methods:

• A modified iterative consensus approach involving a systematic literature review of 216 publications (2023-2025) and analysis of existing standards (e.g., TRIPOD-LLM, CONSORT-AI).
• Development of an initial 44-item checklist by a supervisory group.
• Three rounds of consensus discussions with an 11-member multidisciplinary expert panel, requiring unanimous agreement.

Main Results:

• The final MEDAI-LLM-SUMM checklist includes 24 items across six sections: Clinical validity, Model Selection, Data, Quality Assessment, Safety, and Data Availability.
• The checklist uniquely addresses hallucination assessment, reference summary creation, LLM-as-judge validation, and pilot testing specifications.
• Comparative analysis confirmed MEDAI-LLM-SUMM's superiority over six existing standards in covering essential aspects of medical LLM summarization research.

Conclusions:

• MEDAI-LLM-SUMM provides a comprehensive and specialized framework for reporting medical text summarization research using LLMs.
• Adoption of this checklist can improve the quality, safety, and reproducibility of LLM-based medical summarization studies.
• This standardized reporting will facilitate better clinical deployment readiness and evaluation of LLM technologies in healthcare.