A Dataset for Evaluating Large Language Models on Chinese National Medical Licensing Examinations.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show promise in medical education and clinical reasoning.
• A lack of standardized, non-English datasets hinders LLM evaluation in diverse medical contexts.

Purpose of the Study:

• Introduce CNMLEQA, a novel benchmark dataset for evaluating LLMs on the Chinese National Medical Licensing Examination.
• Address the gap in non-English medical datasets for AI model assessment.

Main Methods:

• CNMLEQA dataset creation by integrating question-answer pairs from PubMed, GitHub, and MedExamLLM.
• Dataset comprises two subsets (CNMLEQA-10k and CNMLEQA-3k) with multiple-choice questions.
• Questions annotated by clinical experts across dimensions: type, year, and clinical scenarios (disease, surgery, medication, lab, symptom).

Main Results:

• Evaluated state-of-the-art LLMs including Gemini, DeepSeek, GPT, Qwen, and LLaMA.
• Qwen2.5-32B achieved 90.88% accuracy on CNMLEQA-10k; DeepSeek-R1 achieved 91.59% on CNMLEQA-3k.
• Fine-tuning experiments demonstrated significant performance improvements for Qwen models.

Conclusions:

• CNMLEQA offers a multidimensional, clinically grounded benchmark for LLM evaluation in Chinese medical applications.
• The dataset facilitates the development and assessment of AI tools for Chinese medical education and practice.