Performance Gaps and Optimization Strategies in Chinese Medical Large Language Models Based on MedBench: Evaluation...

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Medical Informatics

Background:

• Medical large language models (LLMs) require robust evaluation frameworks for safe and accurate clinical deployment.
• Current evaluation methods are insufficient for identifying domain-specific errors and cross-modal challenges in medical LLMs.

Purpose of the Study:

• To develop and apply a granular error taxonomy for systematically identifying weaknesses in leading medical LLMs.
• To create an actionable roadmap for enhancing the clinical robustness, safety, and trustworthiness of medical AI.

Main Methods:

• A granular error taxonomy was developed by analyzing 42,766 responses from 10 top medical LLMs on the MedBench benchmark.
• Incorrect responses were categorized into 8 types: omissions, hallucination, format mismatch, causal reasoning deficiency, contextual inconsistency, unanswered, output error, and deficiency in medical language generation.
• A 4-level tiered optimization strategy was proposed, including prompt engineering, knowledge-augmented retrieval, hybrid neuro-symbolic architectures, and causal reasoning frameworks.

Main Results:

• Analysis of 10 leading medical LLMs revealed significant vulnerabilities despite a 0.86 accuracy in medical knowledge recall.
• Omissions were the most prevalent error type, with a 96.3% omission rate in critical reasoning tasks.
• Safety and ethics evaluations showed low robustness (0.79) under option-shuffled conditions, indicating systemic weaknesses in knowledge boundary enforcement and multistep reasoning.

Conclusions:

• The developed granular error taxonomy provides critical insights into medical LLM vulnerabilities.
• This work establishes a roadmap for enhancing the clinical robustness and trustworthiness of AI in healthcare.
• The findings redefine evaluation paradigms, promoting the responsible application of AI in high-stakes medical environments.