Evaluating Large-Scale and Lightweight Large Language Models for Traditional Chinese Medicine Exam Questions: A Comparative Study | JoVE Visualize

Area of Science:

Artificial Intelligence
Natural Language Processing
Traditional Chinese Medicine

Background:

Traditional Chinese Medicine (TCM) presents unique challenges for large language models (LLMs) due to its knowledge-intensive nature.
While TCM-specific benchmarks exist, the performance of lightweight LLMs in this domain is under-explored.
This study systematically evaluates and compares large-scale and lightweight LLMs for TCM applications, considering deployment trade-offs.

Purpose of the Study:

To systematically evaluate and compare the performance of various large-scale and lightweight LLMs on TCM-related question-answering tasks.
To assess the capabilities and deployment trade-offs of different LLM sizes in the context of TCM.
To identify factors influencing LLM performance, such as prompting strategies and language.

Main Methods:

A dataset of 801 TCM-related questions was developed from textbooks.
Eleven LLMs were evaluated using zero-shot and few-shot prompting in both English and Chinese.
Performance was primarily measured by accuracy, with a focus on different question types and reasoning categories.

Main Results:

Large-scale LLMs excelled in single-choice and true/false questions but underperformed in multiple-choice questions.
Lightweight LLMs generally lagged behind larger models, with Qwen3-1.7B showing competitive performance, even surpassing the specialized TCMChat-7B.
Few-shot prompting and Chinese prompts generally improved LLM performance, while symptomatic diagnosis proved to be the most challenging reasoning task.

Conclusions:

Large-scale LLMs demonstrate strong knowledge recall in TCM but their limitations in multiple-choice tasks and high computational costs may hinder clinical application.
Optimized lightweight LLMs, exemplified by Qwen3-1.7B, suggest that model optimization and domain-specific training are more advantageous than simply increasing model size.
Findings offer insights for deploying optimized LLMs in resource-constrained healthcare environments, though further evaluation in real-world clinical decision-making is warranted.