Evaluating and Improving Syndrome Differentiation Thinking Ability in Large Language Models: Method Development...

Area of Science:

• Artificial Intelligence
• Computational Medicine
• Traditional Chinese Medicine

Background:

• Large language models (LLMs) offer new avenues for the intelligent development of traditional Chinese medicine (TCM).
• Syndrome differentiation is a core component of TCM, and integrating this capability into LLMs is vital for clinical applications.
• The complexity of TCM syndrome differentiation presents a significant challenge for current LLMs.

Purpose of the Study:

• To evaluate the syndrome differentiation thinking capabilities of LLMs.
• To develop and assess a method for enhancing LLM performance in TCM syndrome differentiation.

Main Methods:

• Decomposed TCM syndrome differentiation into pathogenesis inference, syndrome inference, and diagnostic suggestion.
• Created a high-quality evaluation dataset with 200 standardized medical cases.
• Developed an "open-book exam" methodology using customized templates and dynamic knowledge retrieval to generate professional instruction data for fine-tuning LLMs.

Main Results:

• The proposed fine-tuning method significantly improved LLM performance in syndrome differentiation tasks.
• The fine-tuned model achieved 85.7% accuracy in pathogenesis inference and 81.2% in syndrome inference.
• The model's diagnostic suggestions showed 84.3% similarity to expert opinions, outperforming existing general and TCM-specific LLMs.

Conclusions:

• Current LLMs exhibit limitations in TCM syndrome differentiation.
• Fine-tuning LLMs with tailored instruction data and professional knowledge substantially enhances their diagnostic reasoning capabilities.
• Optimized LLMs demonstrate expert-level consistency, offering significant theoretical and practical value for TCM clinical practice.