All That Shines Is Not Gold: Maintaining Scientific Rigor When Evaluating, Interpreting, and Reviewing Studies Using...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Computational Linguistics

Background:

• Large language models (LLMs) are rapidly adopted in healthcare, presenting innovation opportunities.
• This adoption raises critical concerns regarding scientific rigor, transparency, and ethical implications.
• Existing research methodologies may not adequately address the unique challenges posed by LLMs.

Purpose of the Study:

• To provide a comprehensive toolbox for clinicians, researchers, and reviewers engaged with LLM studies.
• To highlight essential methodologic transparency, reproducibility, and ethical considerations in LLM research.
• To offer guidance on navigating the complexities of LLM functioning, application, and evaluation in healthcare.

Main Methods:

• Review foundational aspects of LLM functioning, including training data, biases, and the 'black-box' nature.
• Examine prompt engineering strategies for optimizing model interaction and systematic evaluation.
• Discuss challenges in interpreting LLM outputs, advocating for explainability and fairness.

Main Results:

• Identified key challenges in LLM interpretability, bias mitigation, and ethical deployment.
• Stressed the need for clear reporting of computational resources and environmental impacts.
• Highlighted the risk of rapid model iteration leading to study obsolescence.

Conclusions:

• Traditional peer-review practices are often outpaced by LLM evolution, necessitating new guidelines.
• Rigorous qualitative assessments are required to ensure the validity, fairness, and clinical utility of LLM studies.
• Recommendations are provided to enhance reporting standards and reproducibility in healthcare LLM research.