Guidelines for Reporting Studies on Large Language Models in Radiology: An International Delphi Expert Survey.

Area of Science:

• Radiology
• Artificial Intelligence
• Medical Informatics

Background:

• Large language models (LLMs) show significant potential in radiology for tasks like image analysis and decision support.
• However, a lack of standardized reporting for LLM studies hinders reproducibility and clinical application.

Purpose of the Study:

• To develop standardized reporting guidelines for large language model studies in radiology.
• To enhance the transparency, reproducibility, and clinical applicability of LLM research in the field.

Main Methods:

• A systematic review of 57 LLM studies in radiology (May 2023-March 2024) was performed.
• A two-step process involving a systematic review and a Delphi process with 20 international experts was used.
• A 32-item checklist, FLAIR (Framework for LLM Assessment in Radiology), was created across six key categories.

Main Results:

• The FLAIR checklist comprises 32 items covering data input, prompting, performance, ethics, and implementation.
• Key categories include general information, prompting/fine-tuning, performance metrics, ethics/transparency, and implementation/risks.
• The checklist aims to standardize reporting across diverse LLM applications in radiology.

Conclusions:

• The FLAIR checklist provides a standardized framework for reporting LLM studies in radiology.
• Standardized reporting will foster transparency, reproducibility, and comparability, facilitating clinical translation.
• Adoption of FLAIR can enhance patient care through more reliable AI-driven radiological tools.