Evaluating guideline adherence in LLM studies using LLMs

To evaluate the capability of large language models (LLM), specifically GPT-4 and o1, in assessing adherence to the MI-CLEAR-LLM checklist in previously published studies.

A total of 159 medical research articles related to LLM applications were analyzed. Two models-GPT-4o and o1-were tested in both text-based and image-based modalities. Structured prompts incorporating reasoning strategies such as chain-of-thought and few-shot learning were used to extract information corresponding to the six core items of the MI-CLEAR-LLM checklist. Human evaluations from a prior study served as the reference standard. Each model was evaluated across three independent trials to assess consistency. Accuracy and inter-trial agreement were calculated for each checklist item.

Both GPT-4o and o1 demonstrated high accuracy in extracting objective, explicitly reported items, such as LLM specifications (name, manufacturer, web access, 85.9-100%) and stochasticity parameters (63.6-95%). However, performance declined for context-dependent items, including prompt session handling (Item4, 51.5-70.7%) and test data independence (Item6, 59.6-76.8%). Text-based models generally showed superior inter-trial consistency, with GPT-4o-text achieving the highest Fleiss' kappa (κ = 0.926). In contrast, image-based models exhibited greater variability (κ = 0.402-0.772).

LLMs show strong potential for automating the evaluation of reporting quality in medical research, particularly for clearly structured content. However, they still face substantial challenges in extracting context-dependent or inferential information.