Structured taxonomy and framework for developing medical benchmark in large language models derived from scoping...

Area of Science:

• Medical Informatics
• Artificial Intelligence
• Natural Language Processing

Background:

• Large language models (LLMs) show promise in medicine, necessitating robust evaluation methods.
• Existing benchmark datasets for medical LLMs lack standardized guidance for selection and design.
• Ensuring the reliability and ethical application of LLMs in healthcare is paramount.

Purpose of the Study:

• To propose a structured taxonomy for selecting medical LLM benchmarks.
• To introduce the READY framework for systematic medical benchmark development.
• To enhance the rigor and ethical considerations in evaluating medical LLMs.

Main Methods:

• Systematic literature review of 55 studies on medical LLM benchmarks.
• Analysis of benchmark datasets using a structured framework (dataset construction and evaluation methodology).
• Validation of the proposed framework by five domain experts with consistent inter-rater agreement.

Main Results:

• A structured taxonomy for medical LLM benchmark selection was established.
• The READY framework (Reliable, Ethical, Annotated, Diverse, Yield-validated) was developed.
• Expert validation confirmed the framework's applicability and consistency.

Conclusions:

• The proposed taxonomy and READY framework offer practical guidance for medical LLM benchmark development.
• This research aims to foster more rigorous, ethical, and reliable LLM evaluations in medicine.
• Implementation of these tools will support the safe clinical deployment of LLMs.