Evaluating guideline adherence in LLM studies using LLMs.

Area of Science:

• Medical research reporting standards
• Artificial intelligence in healthcare
• Natural Language Processing (NLP) applications

Background:

• The MI-CLEAR-LLM checklist aims to standardize reporting quality for medical research involving large language models (LLMs).
• Evaluating LLM adherence to reporting guidelines is crucial for transparency and reproducibility in medical studies.
• Previous methods for checklist assessment were manual and time-consuming.

Purpose of the Study:

• To assess the capability of advanced LLMs, specifically GPT-4o and o1, in automatically evaluating adherence to the MI-CLEAR-LLM checklist.
• To compare the performance of text-based versus image-based LLM modalities in this assessment task.
• To determine the consistency and accuracy of LLM-driven checklist evaluations.

Main Methods:

• Analysis of 159 medical research articles focusing on LLM applications.
• Testing GPT-4o and o1 models in both text and image modalities using structured prompts with reasoning strategies.
• Utilizing human evaluations as a reference standard and conducting three independent trials per model for consistency assessment.

Main Results:

• Both GPT-4o and o1 achieved high accuracy (85.9-100%) for explicit LLM specifications and good accuracy (63.6-95%) for stochasticity parameters.
• Performance decreased for context-dependent items like prompt session handling (51.5-70.7%) and test data independence (59.6-76.8%).
• Text-based models demonstrated superior inter-trial consistency (GPT-4o-text: κ=0.926), while image-based models showed greater variability (κ=0.402-0.772).

Conclusions:

• LLMs possess significant potential for automating the assessment of reporting quality in medical research, especially for structured information.
• Challenges remain in LLM performance for extracting context-dependent or inferential reporting details.
• Further refinement of LLMs is needed to improve their ability to critically evaluate complex research reporting elements.