TCM-CoT-RAG: a chain-of-thought enhanced retrieval-augmented generation system for clinical decision support in...

Traditional Chinese Medicine (TCM) rheumatology presents unique challenges for AI-assisted clinical decision support, as the diagnostic process relies heavily on tacit knowledge and individualized reasoning. While Large Language Models (LLMs) have shown promise in medical applications, they remain limited by hallucination risks and inability to replicate expert TCM reasoning. Retrieval-Augmented Generation (RAG) offers a potential solution, yet its application to complex TCM dialectical reasoning remains underexplored.

We developed TCM-CoT-RAG, a hybrid framework combining RAG with Chain-of-Thought (CoT) prompting, grounded in 1,700 expert-curated clinical cases (1,600 for RAG retrieval; 100 for evaluation, including 50 for blinded expert review by three senior TCM rheumatologists). Deployed on Alibaba Cloud, the five system leverages state-of-the-art LLMs (DeepSeek-V3, Qwen3-235B) under a human-in-the-loop paradigm. We designed a dual-tier evaluation: (1) Objective extraction tasks (Task 1-2) quantified using F1-scores; (2) Generative tasks (Task 3-5) assessed using BERTScore. Two senior TCM rheumatologists (≥15 years clinical experience) blindly assessed model outputs, and a senior chief expert quantified consistency between model predictions and ground truth (GT). Comprehensive ablation studies (S1-S4, S-Skip) isolated the contributions of each CoT module.

TCM-CoT-RAG substantially improved diagnostic accuracy across five LLMs. DeepSeek-V3 with full-chain CoT-RAG achieved Entity F1 of 44.89% (+16.45% over baseline) and Formula F1 of 32.13% (+8.74% over baseline), with BERTScore of 0.81 indicating strong semantic alignment with expert reasoning. Ablation confirmed that the complete CoT pipeline was essential-removing any reasoning module caused performance collapse below the zero-shot baseline. Two independent experts validated clinical utility (Cohen's κ > 0.7). DeepSeek-V3 achieved the highest ground-truth consistency at 81.6%, and consistency metrics were quantified by the third expert holding the most senior professional title.

This proof-of-concept framework demonstrates the potential of RAG-enhanced CoT reasoning to improve diagnostic consistency in TCM, objectifying the Symptom-Diagnosis-Prescription pipeline. It is important to note that this system is designed as an AI-assisted clinical decision-support tool. All recommendations require validation by qualified TCM practitioners before clinical application.