Improving TCM question answering through tree-organized self-reflective retrieval with LLMs

Large language models (LLMs) offer significant potential for intelligent question answering (Q&A) in healthcare, yet traditional knowledge representation methods fail to capture the complex, hierarchical nature of Traditional Chinese Medicine (TCM) knowledge systems. The lack of effective retrieval-augmented generation (RAG) frameworks specifically tailored for TCM's unique epistemology limits applications.

This study aims to evaluate the effectiveness of a novel Tree-Organized Self-Reflective Retrieval (TOSRR) framework in enhancing LLM performance on TCM Q&A tasks through innovative knowledge organization and dynamic self-correction mechanisms.

We developed a hierarchical knowledge representation system that structures TCM knowledge as subject-predicate-object-text (SPO-T) units within a tree-like architecture, enabling multi-dimensional relationships while preserving semantic context. Our iterative self-reflection mechanism implements dynamic knowledge retrieval and validation across textbook chapters and disciplines. Performance was evaluated using randomly selected questions from the TCM Medical Licensing Examination (MLE) and college Classics Course Exam (CCE), representing both standardized clinical knowledge and classical theory assessment.

When integrated with GPT-4, the TOSRR framework demonstrated a 19.85% improvement in absolute accuracy on the TCM MLE benchmark and increased recall accuracy from 27 to 38% on CCE datasets. Expert manual evaluation revealed substantial enhancements across critical dimensions: safety, consistency, explainability, compliance, and coherence, with a comprehensive improvement of 18.64 points. Retrieval-Augmented Generation Assessment (RAGAs) metrics confirmed the framework's superior knowledge utilization, retrieval precision, and resistance to information noise compared to standard RAG approaches.

The TOSRR framework enhances LLM performance in TCM knowledge tasks through its hierarchical knowledge representation and self-reflective retrieval approach. And the framework has potential for application in teaching.