A Survey on Medical Competence Evaluation Benchmarks for Large Language Models

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Evaluation

Background:

• Large language models (LLMs) demonstrate significant potential for transforming healthcare applications.
• The critical nature of medical practice necessitates a thorough evaluation of LLM medical competence.
• Existing LLM evaluation methods lack a systematic approach for clinical readiness.

Purpose of the Study:

• To conduct a comprehensive review of methodologies and benchmarks for evaluating LLM medical competence.
• To propose a structured tri-dimensional framework for LLM assessment in healthcare.
• To provide insights into future LLM development and standardization for medical integration.

Main Methods:

• Systematic review of current LLM evaluation practices.
• Analysis of assessment across medical knowledge, clinical practice, and ethical-safety domains.
• Integration of clinician competency assessment frameworks into LLM evaluation.

Main Results:

• Identified established methodologies and benchmarks for LLM medical competence assessment.
• Developed a tri-dimensional framework categorizing evaluations into medical knowledge, clinical practice, and ethical-safety.
• Highlighted gaps in current assessment practices and proposed standardization protocols.

Conclusions:

• A structured, tri-dimensional framework is essential for rigorously evaluating LLM medical competence.
• Standardization protocols are crucial for the safe and effective integration of LLMs into clinical practice.
• Future research should focus on refining evaluation metrics and ensuring ethical deployment of LLMs in healthcare.