Performance Gaps and Optimization Strategies in Chinese Medical Large Language Models Based on MedBench: Evaluation...

The evaluation and improvement of medical large language models (LLMs) are critical for their real-world deployment, particularly in ensuring accuracy, safety, and ethical alignment. However, existing frameworks are inadequate for dissecting domain-specific error patterns or addressing cross-modal challenges.

To address the limitations of current evaluation methods, this study aims to develop and apply a granular error taxonomy to systematically identify critical weaknesses in leading medical LLMs. The ultimate goal is to establish an actionable road map for enhancing their clinical robustness, safety, and overall trustworthiness.

This study introduces a granular error taxonomy developed through a systematic analysis of 10 top-performing models on MedBench (specifically AntAngelMed, Citrus-2.0, INF-Med, WHU_Med, zhuomuniao-Med, TeleChat2, hunyuan-med, UNI-GPT, fusiontech-Med, and GPT-4). Incorrect responses were categorized into 8 distinct types: omissions, hallucination, format mismatch, causal reasoning deficiency, contextual inconsistency, unanswered, output error, and deficiency in medical language generation. Based on these findings, we propose a tiered optimization strategy spanning 4 levels, from prompt engineering and knowledge-augmented retrieval to hybrid neuro-symbolic architectures and causal reasoning frameworks.

Based on a comprehensive error analysis of 42,766 question responses generated by the top 10 models, the evaluation using the 8 defined metrics revealed significant vulnerabilities in the leading models. Despite achieving a high accuracy of 0.86 in medical knowledge recall, analysis across the 8 error categories identified omissions as the most prevalent issue, exhibiting a staggering 96.3% omission rate in critical reasoning tasks. Furthermore, safety and ethics evaluations showed alarming inconsistency under option-shuffled conditions, with a low robustness score of 0.79. Our analysis uncovers systemic weaknesses in the models' ability to enforce knowledge boundaries and perform multistep reasoning.

This work establishes an actionable road map for developing more clinically robust LLMs. By providing error-driven insights, it redefines evaluation paradigms, ultimately advancing the safety and trustworthiness of artificial intelligence in high-stakes medical environments and promoting its responsible application.