Methodological Considerations in Evaluating Large Language Models for Anatomy Education.

Area of Science:

• Medical Education
• Artificial Intelligence
• Anatomy

Background:

• Large language models (LLMs) show potential in anatomy education, clinical decision support, and knowledge assessment.
• Evaluating LLMs with non-English anatomy questions is crucial due to English-dominated datasets.
• Comparative LLM studies face methodological challenges impacting reproducibility, fairness, and interpretation.

Purpose of the Study:

• To highlight critical methodological factors for future large language model evaluation studies.
• To emphasize the need for transparency and standardized reporting in LLM research.
• To ensure the reliability and comparability of LLM assessments in medical fields.

Main Methods:

• Categorization of methodological factors into technical, session-related, and experimental design.
• Identification of specific variables within each category (e.g., model version, prompt wording, scoring).
• Recommendation for adherence to emerging reporting frameworks like TRIPOD-LLM, CONSORT-AI, and MI-CLEAR-LLM.

Main Results:

• Technical factors include model version, interface, access date, browser, device, OS, connection, and timing.
• Session-related factors encompass memory, personalization, chat history, custom instructions, and testing session types.
• Experimental design factors involve dataset source, question selection, prompt wording, input language, response format, attempts, order, scoring, expert evaluation, and inter-rater agreement.

Conclusions:

• Detailed methodological reporting is essential for future LLM comparison studies in anatomy, education, and clinical fields.
• Adopting standardized frameworks will enhance the transparency, reproducibility, and comparability of LLM evaluations.
• Addressing these factors will strengthen the evidence base for LLM applications in healthcare and education.